I actually think we should generally discourage custom CalibrateTasks for individual cameras, and find another way to allow cameras to customize what they need, whether that’s loading PSF models from elsewhere (as in SDSS processing) or finding and repairing CRs at the Snap level instead of at the visit level (for LSST processing). As you’ve noted, the main logic of CalibrateTask is too complex and too important to expect cameras to rewrite the whole thing, and I think the best way to fix that is to have the substitutions happen at a lower (i.e. subtask) level.
I also think we need to take a look at the sketch of the longer-term plan we put together here: Single-frame processing sketch
I’m not sure how much we can move towards that model now, but any steps we can take that won’t cause problems in the short term would be welcome. That would include:
-
Splitting image characterization from Source measurement (breaking up ProcessCcd, and possibly promoting CalibrateTask to a CmdLineTask). We need to do image characterization before we can do anything else, but from there we may want to do other things before we do source measurements (and we frequently may not care about doing source measurement at all).
-
Having two stages of PSF estimation. The first would be more or less like the one we have now, but with no need to use a catalog for star selection, and hence no need for preliminary astronometry - I think that actually simplifies things considerably. The second would be a totally new stand-alone task which would only temporarily use the same algorithms and interfaces, and would use a catalog of stars that isn’t just the external reference catalog (it’d be something we define ourselves in another task, using our own processing results from multiple visits and possibly external catalogs). I think it’d be okay to defer implementing the second one for now, which means this action item might just mean temporarily removing support for CatalogStarSelector and taking advantage of the simplification that offers.
-
Re-detect after determining the initial PSF model instead of reusing the detections from before. This gives us two different icsrc outputs, but it’d remove all the SchemaMapper complexity that’s in the current CalibrateTask, which is a huge win for readability. We should also consider running the deblender between the second stage of detection and the measurement stage that follows, and at least have an option to run it after the first stage of detection.
-
We’d like to do matching to an external catalog (and hence initial astrometry and photometry) over the full visit rather than within individual CCDs. I think we probably want to tackle this after everything else is done, as I’m not sure what parallelization approach we’ll take here, but I don’t think we can call the CalibrateTask overhaul complete until this is done, too.