We have been developing an obs package to process VISTA imaging with the LSST stack. A big question for us has been whether to ingest the community 6 exposure stacks (ISR already performed, deep enough to get good photocal matches and measure ‘PSF’) or the direct exposures (harder to measure PSF because of depth but measurement done as expected by the stack on single exposures, requires a lot of effort to rewrite ISR etc). We have been told that if we use the stacks then we need to ingest the variance images made by the VISTA (Cambridge Astronomical Survey Unit - CASU) pipeline. I have therefore been looking at the ‘instcal’ ingestion and nullISR task in obs_decam.
I can see how the variance and masks are being ingested but as far as I can see the nullIsr task simply puts the ISR ‘instcal’ images after the ISR stage. I don’t see how it is propagating the ‘wtmap’ which has been ingested. What our obs package is currently doing is simply pretending the stacks are exposures and letting it estimate the variance image. Is there a different processCcd task developed somewhere other than the main GitHub repo?
What has been the general experience with the obs_decam and instcal images? Would those with experience recommend using the community pipeline images or is it always better to use the LSST exactly as designed on raw images. Could I see a full example of running the pipeline with instcal images?
Many thanks for everyone’s help over various questions,
It seems that what obs_decam does is “retarget” the nullIsrTask if one wants to use the instcal images (see step 6 in their README file). Retargeting is where one writes a separate bit of code in your python directory, which then overrides a default step. The instruction to retarget is provided in the config file for that task (so, you’d put the snippet of code in step 6 of the readme in config/isr.py).
There are quite a number of retargets in obs_goto (plus I expect most other obs packages), if you want to see some more examples or retargeting.
I see that in retargeting isr to the null isr task all that happens is that it propagates the instcal image to “postISRCCD” stage which presumably inputs to the subsequent stage of processCcd but I don’t see how that incorporates the wtmap. Presumably if we skip all of isr we would have to also produce other products from isr (in particular the variance). I’m also not entirely sure what the “postISRCCD” product is since it isn’t specified in the mapper. I’m also not sure how to run this with processCcd since it needs an id to select the images for processing can I simply add the retarget to the isr config and then run the processCcd command line tasks with the instcal images as inputs. If so why ingest the wtmap and dqmask products into the Butler repo? Or are they exactly the products generated by ISR normally?
And if so do I need to make a wtmap and dqmask plane in the format expected by the stack?
In general I would not recommend modeling anything after how obs_decam handles instcals. The Data Management team is moving away from that with a strong preference for ingesting raw images and calibrations, building our own master-calibs, running ISR, etc. I do not think anything pertaining to how instcals presently work is guaranteed to continue working with the Gen 3 middleware.
For what it’s worth, the current Gen 2 processCcd runs ISR, image characterization (background measurement), and photometric and astrometric calibration. postISRCCD is what comes out of ISR, icExp is what comes out of background characterization, and calexp is what comes out after photometric and astrometric calibration. The new Gen 3 way of doing this will be defining a pipetask which runs each of these tasks in succession (plus whatever other tasks you want!) rather than the monolithic processCcd.
The output from the IsrTask is an lsst.afw.image.Exposure object (with image, mask and variance planes) where counts on the image (roughly) corresponds to astronomical flux. Usually it reads a raw image using the butler, applies various corrections, and sets the mask and variance plane appropriately for CCDs. Your VistaIsrTask (to which you’ll retarget the default isr) needs to have the same result. How you achieve that result is up to you, but you need to respect the interface. I don’t know what the Vista pipeline provides so I can’t make a hard recommendation, but here are some possibilities:
Mask plane:
Looking for magic values in the image plane (e.g., NaN or -999);
Using a bad pixel mask from the Vista pipeline.
Variance plane:
Estimate using metadata (sky brightness, gain, read noise, number of exposures, integration time);
Measure the noise in the background (e.g., with lsst.afw.math.makeStatistics);
Using a variance (or noise) image from the Vista pipeline.
If you want to read something from the Vista pipeline in order to set the mask or variance in your VistaIsrTask, you have to decide whether or not to use the LSST data butler. We think that the data butler is generally helpful, but perhaps it’s an annoying abstraction for you right now. If you don’t use the LSST data butler, you may have more trouble moving up to the Gen3 middleware when that becomes ready, but it might let you get things done quicker right now if you know the file paths to use and read things directly, perhaps using afw primitives or astropy.io.fits or some other I/O code, so long as you’re putting everything into an lsst.afw.image.Exposure object in the end.
I don’t think it’s too hard to use the data butler (though perhaps I’m too familiar with to it to be unbiased). All you need to do to get started is to add an entry to your obs package’s policy/WhateverMapper.yaml file, e.g., for instcal in DecamMapper. The trick is that the filename template is within the butler’s data repo, so the files need to live in the butler’s data repo, with some programmatic directory structure; so you need to get the files in the right place. If you have a suitable layout for your files already, then you can just cp -r the files into the data repo and you’re done. Otherwise, you’re going to need to create the layout. The ingestImages.py script will not work for this (it’s designed to work on raw files, not supplementary files). You could hack that up to install additional files alongside the raw images (that’s what we do for PFS), but possibly the easiest solution would be to write your own script which will install the additional files in the right place in the butler data repo.
I just took a look at the obs_decam ingest code, and it looks a lot more sophisticated than I had thought. Something like that might work for you.
The dqmask and wtmap are being integrated into the instcal dataset by some butler magic rather than being done in their NullIsrTask. I think this is a matter of personal preference (I would have done the construction in their NullIsrTask, but doing it the way it’s done just has different pros and cons).