Is there a reference manual and/or tutorial for using gen3 butler?

Where can I find the most up-to-date information for first-time users for how to ingest and process a RAW FITs source catalog and reference catalog using gen3 butler? Apparently, the information presented here: https://pipelines.lsst.io/v/DM-11077/getting-started/index.html is outdated

1 Like

I’m not aware of any good tutorials for this, but the first steps - setting up a data repository and ingesting the raws - involve the butler create, butler register-instrument, and butler ingest-raws commands. All of those have --help options that can provide some reference information.

One particularly important piece of information for these is the name and class for the instrument. There are actually different instrument classes for different simulators for LSSTCam data (ImSim and PhoSim). If you’re using one of those - and the versions of those we regularly work with - then this will be relatively straightforward. If not, there may be a lot of work involved, either in adding a new Instrument class or modifying the simulator or simulations to closely resemble the output of a supported one.

I am using a Euclid simulation of the LSST e2V sensor following the LsstCam FITs format. This uses a different FITs format than ImSim (which simulates the ITL sensor), and I had issues ingesting and processing with gen2 middleware. I am not sure if I will have issues now with gen3. I am wondering what configurations I might need to set.

If the gen2 problems involved understanding/translating the header metadata, I suspect the problems will remain in Gen3, because they use the same tooling for that.

I think the best option is probably to copy the ImSim or LSSTCam Instrument and translator classes on a branch or fork of obs_lsst to make new ones for this simulator, and modify them as needed until they work. There may be a fair amount of trial-and-error.

Actually, Tim Jenness wrote a new translator for the metadata for LsstCam. I have to use the LsstCam mapper. However, I ran into further problems involving some of the header keywords settings. Kian-Tat gave me a workaround. But, I admit that I don’t quite understand all of the keyword definitions/settings in LSE-400. So, yes it’s been trial and error.

She needs to use lsst.obs.lsst.LsstCam for instrument registration in gen3. The translator seems to be fine. This is discussed elsewhere which also shows the error message from processCcd.py.

Ah, I’m afraid I haven’t caught up yet on the ID mangling and bit-consumption issue; it’s on my to-do list.

I completed the first steps of ingesting the raw FITS, but what is the command for ingesting the reference catalog? And, what is the command to process ccd?

We don’t actually have any tooling for native ingestion of reference catalogs into gen3 yet; all of our reference catalogs are already in gen2 repos, and we have just been using our gen2->gen3 conversion tools to get them into gen3 repos. So I’m afraid there’s more development needed there as well.

You should be able to perform the first steps of processing without a reference catalog - that would be invoking the pipetask tool (from ctrl_mpexec) on just two tasks (“isr” and “characterizeImage”) from our DRP pipelines. I don’t remember the syntax off the top of my head (others following along might), but first I think you’ve got another problem: you’ll need some calibration data - probably at least flats and biases to keep detection and measurement from falling over, though there are many others that are important for getting high-quality results. Do you have raw calibration frames you plan to combine to make masters, or some other way of obtaining calibrations?