How to construct calibs in Gen3?

Support
#1

Hi,

I would like to make calib files using ingested raw data. Previously (using Gen2 Butler) this was accomplished with constructCalibs.py. How can I achieve the same thing in Gen3?

Thanks!

#2

I have just come across these pipelines in obs_lsst, which seem to be the new way of doing this:

#3

Follow-up question:

How can I use Gen3 butler to query raw files with observation_type="bias"? My current attempt looks like this:

In [37]: registry.queryDatasets("raw", collections="Huntsman/raw/all", where="observation_type=bias")
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_butler/21.0.0+187b78b4b8/python/lsst/daf/butler/registry/queries/expressions.py in categorizeElementId(universe, name)
    155         try:
--> 156             dimension = universe[table]
    157         except KeyError as err:

/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_butler/21.0.0+187b78b4b8/python/lsst/daf/butler/core/dimensions/_universe.py in __getitem__(self, name)
    165     def __getitem__(self, name: str) -> DimensionElement:
--> 166         return self._elements[name]
    167

/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_butler/21.0.0+187b78b4b8/python/lsst/daf/butler/core/named.py in __getitem__(self, key)
    329         if isinstance(key, str):
--> 330             return self._mapping[key]
    331         else:

KeyError: 'observation_type'
#4

Yes, cp_pipe is the package dedicated to calibration production. Note that once you have created the calibrations you will need to certify them into a CALIBRATION collection (using something like butler certify-calibrations).

The observation_type is part of the exposure record. You need to do something like: where="exposure.observation_type = 'bias'" (note the quoting of the string “bias” there). You can constrain your query with other parts of the exposure record as well (you can see the options by doing butler query-dimension-records REPO exposure).

1 Like
#5

Hi @timj , thanks for your response. I am having some trouble running the constructBias pipeline:

export REPO=/opt/lsst/software/stack/br
butler create $REPO
butler register-instrument $REPO lsst.obs.huntsman._instrument.HuntsmanCamera
butler ingest-raws -t symlink $REPO /opt/lsst/software/stack/huntsman-drp/tests/data/raw/bias_0.fits
butler query-datasets $REPO raw  # This shows the ingested raw file

# Run the pipeline (identical to the one in cp_pipe)
pipetask run -d "instrument='Huntsman' and exposure=210107013235403" -b ${REPO} -i Huntsman/raw/all -o calibs_temp/bias -p ${OBS_HUNTSMAN}/pipelines/constructBias.yaml

This results in the error:

FileNotFoundError: Not enough datasets (0) found for non-optional connection isr.camera (camera) with minimum=1 for quantum data ID {instrument: 'Huntsman', detector: 13, exposure: 210107013235403, ...}.

Any ideas? I am running the latest stack release via docker. Thanks!

#6

Your error also occurs in this topic: Issues with ingestCalib.py using DECAM images - #10 by timj

:slight_smile: These kinds of synergies are why we’re using Community for support.

#7

Hi @ktl

Thanks for pointing that out! The solution there (writing curated calibrations) worked. Although in my case, I didn’t actually have any curated calibrations to write. Perhaps the solution was to create a CALIBRATION collection, which happens automatically when using the write-curated-calibrations cli.

#8

It will always write the camera calibration object.