Today I merged multiple PRs for DM-27170, which officially deprecates the
lsst.afw.image.Filter interface, for removal after v22 is released. This is an accelerated deprecation schedule from that described in the dev guide, following RFC-748. This is the next stage of the work described by @kfindeisen in Migrating from afw.image.Filter to FilterLabel.
Feel free to ask questions here about these changes: I’ve attempted to explain some of the details below, but I’m sure I haven’t caught everything relevant.
Physical and Band filter labels
With this change, the inputs and outputs of various tasks will have specific kinds of filter label associated with them. For example, refcat filterMap input configs and coadd outputs are identified by
bandLabel, whereas the visit/ccd
"filter" keys in the
CoaddInputRecorder output table are now explicitly
physicalLabel. In most cases, this change will be entirely transparent to the user:
Exposures store the
FilterLabel, which–except for coadds which are now band only–contains both
bandLabel, and user-specified dataIds should not change (see below).
I think the biggest change will be for HSC coadd postprocessing output parquet tables, which now have a
band MultiIndex instead of
filter, which now has the
bandLabel instead of whatever was in
dataId.get('filter'). This likely requires a change to pipe_analysis and/or to notebooks that access these parquet files.
@kfindeisen and I have tried to maintain backwards compatibility with data processed prior to DM-27169 (weekly
w_2020_50), but we cannot guarantee that all older data files will have fully-populated FilterLabels containing both band and physical labels. Please file tickets to report problems with older processed data and we will try to update the logic.
I believe that these changes will not have any effect on how users specify gen2 dataIds to Tasks. I have attempted to clean up misuses of
dataId['filter'] in task code where I could, but I may not have found all the relevant ones. No changes were required to any of the dataIds specified in ci_hsc_gen2.
dataId['filter'] was always ambiguous in the past–so long as you were self-consistent throughout processing and matched the dataset template expectation, you could use whatever name there you wanted–and that continues to be true. The values that are stored in the output data products are now more explicit, and may not match what was specified in
dataId['filter'], which did not exist for all data products/cameras anyway: for example, DECam
calexps have never had
'filter' in their templates, and thus not in their dataIds.
If you need to record or access the label of a filter associated with a particular set of processing, get the
phyiscalLabel of your
FilterLabel. Code that previously got a filter name from the dataId via e.g.
dataId["filter"] should instead get the
FilterLabel from the associated exposure (
deepCoadd, etc.) and then use either the
physicalLabel as appropriate, for example:
butler.get("calexp_filterLabel").bandLabel. An example of such a change is in pipe_tasks multiBandUtils.readCatalog().
Gen3 dataIds do not have a concept of “filter”. Instead there are
physical dimensions in a gen3 dataId, in addition to
instrument. Because the gen3 registry understands the relationship between these dimensions, users can query the
instrument dimensions and the middleware will fill in the correct physical filter in the returned dataId, when it is necessary for the task.
The equivalent gen3 code to get the
FilterLabel attached to an exposure is
butler.get("calexp.filterLabel"); note the underscore replaced by a period compared with the gen2 version above.
Unfortunately, not all of the changes necessary to allow us to remove
Filter from the stack happened on DM-27170; some required domain expertise that I did not have. The following tickets describe the remaining conversion work:
- DM-28093: IsrTask should use physicalLabel everywhere
- DM-28166: How to compute Coadd exposureIds without
- DM-28334: HSC’s makeTransmissionCurves should use physicalLabels
- DM-28503: sdm_schemas should be updated to refer to “band” instead of “filter”
- DM-28088: Update fgcmcal to use FilterLabel