Gen3 API changes and configuring reference catalogs

Data Management DM Notifications
#1

I’ve just merged DM-17023, a major refactoring of Gen3 Butler APIs that includes synchronized changes for many packages. Developers not working on Gen3 middleware should not see any code they’re using change, but as with any multi-package ticket, it’ll make it easy to get package inconsistency problems when building against the latest weekly. As usual, to avoid that, branch from the weekly tag when building against the weekly, and rebase on master when running via Jenkins or lsstsw.

This change does affect Gen2 (and hence most Science Pipelines developers) in one significant way: CmdLineTask/PipelineTask hybrids that rely on a reference catalog (CalibrateTask and MeasureMergedCoaddSourcesTask, to be specific) now have different configuration options for specifying reference catalog names in Gen2 and Gen3 (due to structural differences in how those catalogs are loaded). To avoid subtle, silent bugs later, we also require that those options be set consistently when running with either middleware generation. This obs_lsst commit shows the kinds of changes that involves. Such changes have already been made on all relevant obs package config overrides, but will also need to be made to any external config files that modify the reference catalog.

1 Like
#2

What about other reference catalog options (e.g., calibrate.astromRefObjLoader.filterMap)? If (by the sound of it) refObjLoader is going away in Gen 3, how do we configure reference catalog details?

#3

I think the RefObjLoaders (and their config options) will be pared down but not removed when we retire Gen2, and hence we’ll continue a bit longer having additional options like filterMap that also have to be kept in sync. Eventually I think we need to rethink a lot of how we map reference catalog filters to data-being-processed filters, exchanging color terms for inferred SEDs and integrals over transmission curves. But I don’t have anything like a complete design for that in mind yet, and I hope to keep that as independent as possible from the middleware changes.

#4

There are some sketches of how a possible replacement might look in this post: Reference catalogs, camFluxes, and colorterms

I think the end result of that discussion was that we should wait for Gen3 to land before we rework all of that.