How to see the Config params used for a task (e.g., deblender)

Support Data Preview 0
#1

The scenario is that a user wants to know the input configuration parameters that the deblender used. I have two questions about how to find that information.

(1) We can do the following, and see the contents of the configuration class, but I suspect that this code is printing the default configuration parameters for Weekly 2023_07 of the LSST Science Pipelines, not necessarily what was run to create the DP0.2 data products. Is that a correct interpretation?

from lsst.meas.deblender import SourceDeblendTask
import lsst.afw.table as afwTable
schema = afwTable.SourceTable.makeMinimalSchema()
sourceDeblendTask = SourceDeblendTask(schema=schema)
help(sourceDeblendTask.ConfigClass)

Output:

Help on class SourceDeblendConfig in module lsst.meas.deblender.sourceDeblendTask:

class SourceDeblendConfig(lsst.pex.config.config.Config)
 |  SourceDeblendConfig(*args, **kw)
 |  
 |  Method resolution order:
...

(output truncated, lots of info, but it gives a whole list).

(2) We can also use the butler to get DP0.2 deblender data products, and we can see that there is a deblend_config in the butler:

import lsst.daf.butler as Butler
butler = Butler.Butler('dp02', collections='2.2i/runs/DP0.2')
dataId = {'tract': 2897, 'patch': 5, 'band': 'i'}
registry = butler.registry
for dt in sorted(registry.queryDatasetTypes('*blend*')):
    print(dt)

But whereas these two butler.get statements work:

dbFlux_schema = butler.get('deepCoadd_deblendedFlux_schema', dataId=dataId)
dbFlux = butler.get('deepCoadd_deblendedFlux', dataId=dataId)

This one returns an error:

dbConfig = butler.get('deblend_config', dataId)

Getting the configuration parameters with a statement like the above is something that has worked for me in the past, but in a scenario where I’ve created a simpleButler and rerun aspects of the pipelines, and then asked for the configs of what I’ve rerun (this was in now-defunct tutorials 09a and 09b, creating custom coadds).

Should this work to get the deblender configuration parameters that were used to create the butler-available DP0.2 data products?

1 Like
#2

As far as I can tell, it looks like a configuration deprecation detailed in [DM-33925] Remove loggingInterval fields from tasks - Jira is causing your error in this case.

When using w_2023_07 of the Science Pipelines on the IDF and attempting your above commands I confirm that I return this error:

ValueError: Failure from formatter 'lsst.obs.base.formatters.pexConfig.
PexConfigFormatter' for dataset 2f1817eb-5146-454a-ac40-72d773a27ed1
(deblend_config from s3://butler-us-central1-panda-dev/dc2/2.2i/runs/
DP0.2/v23_0_1/PREOPS-905/step3_32/20220407T030512Z/deblend_config/
deblend_config_2_2i_runs_DP0_2_v23_0_1_PREOPS-905_step3_32_20220407T0305
12Z.py): lsst.meas.extensions.scarlet.scarletDeblendTask.
ScarletDeblendConfig has no attribute loggingInterval

This error is being caused by a now-deprecated config option (loggingInterval) which was in-place when the 2.2i/runs/DP0.2 collection was initially written but now no longer exists.

If instead I switch to an older version of the Science Pipelines, e.g., release R24.0.0, then your above code snippet succeeds as expected (with a FutureWarning: Config field multibandDeblend.loggingInterval is deprecated: This field is no longer used and will be removed in v25.).

Therefore, in cases such as this, the configuration for this task can still be accessed by loading up an older version of the Science Pipelines. Also pinging here @erykoff and @arunkannawadi to ask if you would agree with this recommendation, or if any other course of action may be best instead?

1 Like
#3

Apologies for the delay in responding. I’d rather not have to switch to an older version of the Science Pipelines just to be able to read in something. From what I understand, because of the deprecation and eventual removal of a config parameter, the older configs cannot be read, but the dataset themselves can be?

#4

Yes, I think that’s correct. I wasn’t sure if there’s another way to read in now-deprecated configs other than reverting to an older version of the Science Pipelines?

#5

I don’t think so either. The way I see it, the configs are much more closely related to the Science Pipelines than various datasets are. A set of config parameters from one version is not guaranteed to be compatible with another, especially if they are separated by two or more major versions.

1 Like