Error while trying to create master bias frames

demetradecicco · June 14, 2022, 11:06pm

Hi, I want to ingest DECam images into the LSST pipeline. I installed version v23_0_1 on an x86_64 machine, and managed to ingest a set of raw and calibration frames that I chose as a test sample, but I am having issues when I try to create master bias frames.
I ran this line:

pipetask run -b $REPO -p $CP_PIPE_DIR/pipelines/cpBias.yaml -i DECam/raw/all,DECam/calib -o DECam/calib/on20181217_bias -d "instrument='DECam' AND exposure.observation_type='zero' AND exposure.day_obs=20181217" -j 3 -c isr:overscan.fitType='MEDIAN_PER_ROW' --register-dataset-types

and I got the following error message :

ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 122 quanta for 2 tasks, graph ID: '1655246490.30904-16514'
py.warnings WARNING: /home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g6b22db343a+d18c45d440/python/lsst/daf/butler/registry/databases/sqlite.py:444: SAWarning: Class _Ensure will not make use of SQL compilation caching as it does not set the 'inherit_cache' attribute to ``True``.  This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions.  Set this attribute to True if this object can make use of the cache key generated by the superclass.  Alternatively, this attribute may be set to False which will disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
  return connection.execute(_Ensure(table), rows).rowcount

timer.lsst.daf.butler.datastores.fileDatastore ERROR: Reading from location file:///home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz  with formatter lsst.obs.decam.rawFormatter.DarkEnergyCameraRawFormatter: Took 0.4018 seconds
timer.lsst.daf.butler.datastores.fileDatastore ERROR: Reading from location file:///home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz  with formatter lsst.obs.decam.rawFormatter.DarkEnergyCameraRawFormatter: Took 0.4109 seconds
ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'isr' on quantum {instrument: 'DECam', detector: 7, exposure: 804070, ...} failed. Exception ValueError: Failure from formatter 'lsst.obs.decam.rawFormatter.DarkEnergyCameraRawFormatter' for dataset aa1de9a0-d869-526b-bfe1-faf8794c49c1 (raw from file:///home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz): 
  File "src/image/ImageBaseFitsReader.cc", line 188, in ndarray::Array<T, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray(const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin, bool) [with T = int]
    cfitsio error (/home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz) : Incompatible type for FITS image: on disk is float32 (HDU 7), in-memory is int32. Read with allowUnsafe=True to permit conversions that may overflow.
cfitsio error stack:
 {0}
lsst::afw::fits::FitsTypeError: 'cfitsio error (/home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz) : Incompatible type for FITS image: on disk is float32 (HDU 7), in-memory is int32. Read with allowUnsafe=True to permit conversions that may overflow.
cfitsio error stack:
'

ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'isr' on quantum {instrument: 'DECam', detector: 9, exposure: 804070, ...} failed. Exception ValueError: Failure from formatter 'lsst.obs.decam.rawFormatter.DarkEnergyCameraRawFormatter' for dataset 4a739bd7-6d99-5640-adc9-3bc8c7aa3383 (raw from file:///home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz): 
  File "src/image/ImageBaseFitsReader.cc", line 188, in ndarray::Array<T, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray(const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin, bool) [with T = int]
    cfitsio error (/home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz) : Incompatible type for FITS image: on disk is float32 (HDU 9), in-memory is int32. Read with allowUnsafe=True to permit conversions that may overflow.
cfitsio error stack:
 {0}
lsst::afw::fits::FitsTypeError: 'cfitsio error (/home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz) : Incompatible type for FITS image: on disk is float32 (HDU 9), in-memory is int32. Read with allowUnsafe=True to permit conversions that may overflow.
cfitsio error stack:
'

Process task-25:
Traceback (most recent call last):
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g6b22db343a+d18c45d440/python/lsst/daf/butler/datastores/fileDatastore.py", line 1221, in _read_artifact_into_memory
    result = formatter.read(component=getInfo.component if isComponent else None)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/obs_base/gb0fc2ca601+a0cf348625/python/lsst/obs/base/formatters/fitsExposure.py", line 89, in read
    return self.readFull()
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/obs_base/gb0fc2ca601+a0cf348625/python/lsst/obs/base/_fitsRawFormatterBase.py", line 356, in readFull
    exposure = lsst.afw.image.makeExposure(lsst.afw.image.makeMaskedImage(self.readImage()))
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/obs_decam/gd060f87c5c+a369eca8c2/python/lsst/obs/decam/rawFormatter.py", line 140, in readImage
    return lsst.afw.image.ImageI(self.fileDescriptor.location.path, index)
lsst.pex.exceptions.Exception: 
  File "src/image/ImageBaseFitsReader.cc", line 188, in ndarray::Array<T, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray(const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin, bool) [with T = int]
    cfitsio error (/home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz) : Incompatible type for FITS image: on disk is float32 (HDU 7), in-memory is int32. Read with allowUnsafe=True to permit conversions that may overflow.
cfitsio error stack:
 {0}
lsst::afw::fits::FitsTypeError: 'cfitsio error (/home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz) : Incompatible type for FITS image: on disk is float32 (HDU 7), in-memory is int32. Read with allowUnsafe=True to permit conversions that may overflow.
cfitsio error stack:
'


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ddecicco/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/ddecicco/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/ctrl_mpexec/g6727979600+15d2600a0d/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 129, in _executeJob
    quantumExecutor.execute(taskDef, quantum, butler)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/ctrl_mpexec/g6727979600+15d2600a0d/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 174, in execute
    self.runQuantum(task, quantum, taskDef, butler)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/ctrl_mpexec/g6727979600+15d2600a0d/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 464, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/ip_isr/g32debb59f0+8df549a9bd/python/lsst/ip/isr/isrTask.py", line 985, in runQuantum
    inputs = butlerQC.get(inputRefs)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/pipe_base/g5c83ca0194+970dd35637/python/lsst/pipe/base/butlerQuantumContext.py", line 137, in get
    val = self._get(ref)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/pipe_base/g5c83ca0194+970dd35637/python/lsst/pipe/base/butlerQuantumContext.py", line 92, in _get
    return self.__butler.getDirect(ref)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g6b22db343a+d18c45d440/python/lsst/daf/butler/_butler.py", line 974, in getDirect
    return self.datastore.get(ref, parameters=parameters)
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g6b22db343a+d18c45d440/python/lsst/daf/butler/datastores/fileDatastore.py", line 1846, in get
    return self._read_artifact_into_memory(getInfo, ref, isComponent=isComponent,
  File "/home/ddecicco/lsst_stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g6b22db343a+d18c45d440/python/lsst/daf/butler/datastores/fileDatastore.py", line 1223, in _read_artifact_into_memory
    raise ValueError(f"Failure from formatter '{formatter.name()}' for dataset {ref.id}"
ValueError: Failure from formatter 'lsst.obs.decam.rawFormatter.DarkEnergyCameraRawFormatter' for dataset aa1de9a0-d869-526b-bfe1-faf8794c49c1 (raw from file:///home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz): 
  File "src/image/ImageBaseFitsReader.cc", line 188, in ndarray::Array<T, 2, 2> lsst::afw::image::ImageBaseFitsReader::readArray(const lsst::geom::Box2I&, lsst::afw::image::ImageOrigin, bool) [with T = int]
    cfitsio error (/home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz) : Incompatible type for FITS image: on disk is float32 (HDU 7), in-memory is int32. Read with allowUnsafe=True to permit conversions that may overflow.
cfitsio error stack:
 {0}
lsst::afw::fits::FitsTypeError: 'cfitsio error (/home/ddecicco/lsst_stack/DATA/DECam/raw/all/raw/20181217/ct4m20181217t193441/raw_DECam_solid_plate_0_0_0_0_ct4m20181217t193441_S29_DECam_raw_all.fits.fz) : Incompatible type for FITS image: on disk is float32 (HDU 7), in-memory is int32. Read with allowUnsafe=True to permit conversions that may overflow.
cfitsio error stack:
'

which is repeated for various process tasks and HDU.
Can anybody help me, please? Thanks a lot.

timj · June 14, 2022, 11:18pm

I’m not sure what’s going on there with the type conversion. We have processed DECam data from scratch before:

@lskelvin is it possible that the raw format changed from int to float at some point (as happened at MegaCam when they started to add a BZERO/BSCALE)?

demetradecicco · June 15, 2022, 6:12pm

Indeed I am using this notebook as a refence, Lee kindly shared it with me last week. The line I posted is slightly different than the corresponding line in the notebook, but I get the same error message with both.

timj · June 15, 2022, 8:07pm

The issue is whether raw DECam data changed from int to float at some point and we’ve never encountered this before. I know nothing about DECAm raw files. The reader we use for raws assumes it’s integer.

lskelvin · June 16, 2022, 5:42pm

Hi @demetradecicco, thank you for sharing your full error traceback. For my part, I haven’t seen this error crop up before when processing DECam data from raw files. The data I’ve been working with was taken on DECam within the last several months, so I don’t think they’ve changed their storage format since then. Similarly, @mrawls has also been successfully processing DECam data taken from 2014/2015, which also has the same format.

Can I ask where your raw data comes from originally? Is it direct from DECam, or has it been processed via another pipeline, e.g., the Community Pipeline? If possible, would you be able to make some of your raw data available so that we can take a closer look at this and try to reproduce the error? Please feel free to drop me a line on Slack about that, if easier!

lskelvin · June 23, 2022, 8:15pm

Just to follow up on this thread, in case it is of use to others. Having discussed this issue in more detail with @demetradecicco, it appears that the bias data which was initially ingested using butler ingest-raws was not actually raw data but rather master bias files (PROCTYPE=MasterCal) constructed by the DECam Community Pipeline. This discrepancy is likely the root cause of the errors reported in this thread. At present, the LSST Science Pipelines does not support use of calibration files produced by external data processing pipelines.

The solution in this case would be to download the raw bias frames instead, and then use the $CP_PIPE_DIR/pipelines/DarkEnergyCamera/cpBias.yaml pipeline to construct DECam master bias frames (note that this specialist DECam bias pipeline now exists for DECam master bias construction). The same recommended raws-to-master-calibs process will also hold true for master flat frame construction.

We are looking at adding support for use of master calibration files produced via other pipelines, and may add that functionality into the Science Pipelines in the future. We’re also looking at adding additional functionality which will warn when using butler ingest-raws to ingest a file which is not actually a raw file. Hopefully, that should help catch issues similar to this in the future.