I am trying to ingest some Megacam data. The images have been processed through Stephen Gwyn’s Megapipe, which drops all of the Elixir specific keywords, including IMRED_MK which specifies the appropriate defects mask file. This causes ingestImages to barf.
Is there a way to specify a missing keyword:value pair in a config option to ingestImages?
I’d like to preserve the original images if possible.
I think you are using Gen2 middleware for your processing? Gen2 has been deprecated for over a year and has recently been completely removed.
Gen3 is a significant upgrade but we have not spent a lot of time making it work for megaprime. The fundamentals are there in that we can ingest raw data and the instrument is known to gen3.
We don’t really process much Megaprime data though so there are some big gaps in our gen3 handling waiting for a motivated person to fix. For example:
None of this work is difficult, it’s just never been prioritized.
If you are wanting to ingest processed images you have a slightly different problem in gen3 in that you have to work out the dataId that you want to associate with the data. This will depend on whether it’s single exposures or deep coadds. You would have to use the butler ingest-files command to ingest them and specify the dataIds in a CSV file. A lot of the answer depends on what skymap you are using and whether you would need to teach the butler repository about individual exposures first.
cc/ @yusra
You hit the nail on the head regarding gen2. This work is currently being done in v19 because the various folks (including myself) haven’t finished the gen3 alterations to obs_cfht yet. getting those upgrades done is definitely a priority for me, but I’m just one user. heh.
Anyways, your response indirectly affirms my expectation that I will just have to manually put the keyword back. Not difficult, just another thing to keep track of. Thanks @timj
You are working on gen3 obs_cfht? Are you coordinating that work with us? I’d hate for us to start plugging away on some obs_cfht cleanups and then find you’ve done it all somewhere else. cc/ @czw
I’m trying to remember what ingestImages.py actually did. I think ingestImages did call the fix_header method in astro_metadata_translator so in theory you could override the fix_header method in the MegaPrimeTranslator class to put the headers back automatically. More importantly v19 is from 2019 I think so that’s very old knowledge to dredge up and I’m not even sure we called fix_header back then.
That’s a good suggestion. Thanks. I was thinking of bodging translate_defects.
regarding my work, I am still trying to sort out the differences between gen2 and gen3 on the time I’ve been able to look at this, which admittedly hasn’t been much. I do intend to work with people once I feel I’ve got the knowledge to do so.
FWIW I was able to modify the obs_cfht branch of Dom B. to include the ear chips active at CFHT. In case that hasn’t been done elsewhere.
Recent paper on gen3: [2206.14941] The Vera C. Rubin Observatory Data Butler and Pipeline Execution System
Talk from SPIE meeting in the summer: The Vera C. Rubin Observatory Data Butler and pipeline execution system