Today I merged multiple PRs for DM-27170, which officially deprecates the lsst.afw.image.Filter interface, for removal after v22 is released. This is an accelerated deprecation schedule from that described in the dev guide, following RFC-748. This is the next stage of the work described by @kfindeisen in Migrating from afw.image.Filter to FilterLabel.
Feel free to ask questions here about these changes: I’ve attempted to explain some of the details below, but I’m sure I haven’t caught everything relevant.
Physical and Band filter labels
With this change, the inputs and outputs of various tasks will have specific kinds of filter label associated with them. For example, refcat filterMap input configs and coadd outputs are identified by bandLabel, whereas the visit/ccd "filter" keys in the CoaddInputRecorder output table are now explicitly physicalLabel. In most cases, this change will be entirely transparent to the user: Exposures store the FilterLabel, which–except for coadds which are now band only–contains both physicalLabel and bandLabel, and user-specified dataIds should not change (see below).
I think the biggest change will be for HSC coadd postprocessing output parquet tables, which now have a band MultiIndex instead of filter, which now has the bandLabel instead of whatever was in dataId.get('filter'). This likely requires a change to pipe_analysis and/or to notebooks that access these parquet files.
Backwards compatibility
@kfindeisen and I have tried to maintain backwards compatibility with data processed prior to DM-27169 (weekly w_2020_50), but we cannot guarantee that all older data files will have fully-populated FilterLabels containing both band and physical labels. Please file tickets to report problems with older processed data and we will try to update the logic.
dataIds
gen2
I believe that these changes will not have any effect on how users specify gen2 dataIds to Tasks. I have attempted to clean up misuses of dataId.get('filter') and dataId['filter'] in task code where I could, but I may not have found all the relevant ones. No changes were required to any of the dataIds specified in ci_hsc_gen2.
Gen2 dataId['filter'] was always ambiguous in the past–so long as you were self-consistent throughout processing and matched the dataset template expectation, you could use whatever name there you wanted–and that continues to be true. The values that are stored in the output data products are now more explicit, and may not match what was specified in dataId['filter'], which did not exist for all data products/cameras anyway: for example, DECam calexps have never had 'filter' in their templates, and thus not in their dataIds.
If you need to record or access the label of a filter associated with a particular set of processing, get the bandLabel or phyiscalLabel of your Exposure's FilterLabel. Code that previously got a filter name from the dataId via e.g. dataId["filter"] should instead get the FilterLabel from the associated exposure (calexp, deepCoadd, etc.) and then use either the bandLabel or physicalLabel as appropriate, for example: butler.get("calexp_filterLabel").bandLabel. An example of such a change is in pipe_tasks multiBandUtils.readCatalog().
gen3
Gen3 dataIds do not have a concept of “filter”. Instead there are band and physical dimensions in a gen3 dataId, in addition to instrument. Because the gen3 registry understands the relationship between these dimensions, users can query the band and instrument dimensions and the middleware will fill in the correct physical filter in the returned dataId, when it is necessary for the task.
The equivalent gen3 code to get the FilterLabel attached to an exposure is butler.get("calexp.filterLabel"); note the underscore replaced by a period compared with the gen2 version above.
Next steps
Unfortunately, not all of the changes necessary to allow us to remove Filter from the stack happened on DM-27170; some required domain expertise that I did not have. The following tickets describe the remaining conversion work:
- DM-28093: IsrTask should use physicalLabel everywhere
- DM-28166: How to compute Coadd exposureIds without
Filter.getId() - DM-28334: HSC’s makeTransmissionCurves should use physicalLabels
- DM-28503: sdm_schemas should be updated to refer to “band” instead of “filter”
- DM-28088: Update fgcmcal to use FilterLabel