Problem ingesting/applying DECam calibrations generated from raw data

rkotulla · February 2, 2021, 7:07pm

Hi,

I’m trying to reduce some DECam data from scratch. Data is from the NOAO Science archive, and includes a full night worth of data, including raws for science and calibrations.

For some platform background, I’ve tried this with both release 21 and the w_2021_2 versions, both with identical results (or lack thereof). In terms of packages, I’ve setup obs_decam, pipe_tasks and pipe_drivers.

Ingesting the raws (via ingestImagesDecam.py) and the curated calibration products (defects & crosstalk, via ingestCuratedCalibs.py) worked just fine, as did the constructBias.py step. However, when running the ingestCalibs step I get the following:

ingestCalibs.py . --calib=CALIB/ --rerun=calib --mode=copy --validity=350 rerun/calib/BIAS/2015-08-17/*.fits

root INFO: Loading config overrride file '/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/obs_decam/21.0.0+ed48dff28b/config/ingestCalibs.py'
CameraMapper INFO: Loading exposure registry from /sas3/projects/hydra/decam_test/registry.sqlite3
CameraMapper INFO: Loading calib registry from /sas3/projects/hydra/decam_test/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /sas3/projects/hydra/decam_test/CALIB/calibRegistry.sqlite3
ingestCalibs.register INFO: Table "bias" exists.  Skipping creation
ingestCalibs.register INFO: Table "flat" exists.  Skipping creation
ingestCalibs.register INFO: Table "fringe" exists.  Skipping creation
ingestCalibs.register INFO: Table "dark" exists.  Skipping creation
ingestCalibs.register INFO: Table "illumcor" exists.  Skipping creation
ingestCalibs.parse WARN: Error reading rerun/calib/BIAS/2015-08-17/BIAS-2015-08-17-01.fits extensions {'S5', 'S11', 'N19', 'N7', 'N17', 'S1', 'S18', 'N4', 'N24', 'S6', 'N14', 'S23', 'N27', 'S21', 'N10', 'N29', 'S15', 'N12', 'N18', 'S28', 'N22', 'N31', 'N11', 'N9', 'N2', 'N26', 'N13', 'N1', 'S14', 'N8', 'N25', 'S16', 'S25', 'S24', 'S13', 'S17', 'S20', 'S10', 'S7', 'S26', 'S3', 'N5', 'N6', 'S9', 'N20', 'N23', 'S19', 'S12', 'N3', 'N16', 'S22', 'S27', 'S31', 'S4', 'S8', 'N21', 'N28', 'S2', 'N15'}: 
  File "src/fits.cc", line 529, in void lsst::afw::fits::Fits::setHdu(int, bool)
    cfitsio error (rerun/calib/BIAS/2015-08-17/BIAS-2015-08-17-01.fits): tried to move past end of file (107) : Moving to HDU 5
cfitsio error stack:
 {0}
lsst::afw::fits::FitsError: 'cfitsio error (rerun/calib/BIAS/2015-08-17/BIAS-2015-08-17-01.fits): tried to move past end of file (107) : Moving to HDU 5
cfitsio error stack:
'

... repeats a bunch of time for each frame ...

ingestCalibs INFO: rerun/calib/BIAS/2015-08-17/BIAS-2015-08-17-60.fits --<copy>--> /sas3/projects/hydra/decam_test/CALIB/BIAS/2015-08-17/BIAS-2015-08-17-60.fits
ingestCalibs.parse WARN: Error reading rerun/calib/BIAS/2015-08-17/BIAS-2015-08-17-62.fits extensions {'S5', 'S11', 'S29', 'N19', 'N7', 'N17', 'S1', 'S18', 'N4', 'N24', 'S6', 'N14', 'S23', 'N27', 'S21', 'N10', 'N29', 'S15', 'N12', 'N18', 'S28', 'N22', 'N11', 'N9', 'N2', 'N26', 'N13', 'N1', 'S14', 'N8', 'N25', 'S16', 'S25', 'S24', 'S13', 'S17', 'S20', 'S10', 'S7', 'S26', 'S3', 'N5', 'N6', 'S9', 'N20', 'N23', 'S19', 'S12', 'N3', 'N16', 'S22', 'S27', 'S31', 'S4', 'S8', 'N21', 'N28', 'S2', 'N15'}: 
  File "src/fits.cc", line 529, in void lsst::afw::fits::Fits::setHdu(int, bool)
    cfitsio error (rerun/calib/BIAS/2015-08-17/BIAS-2015-08-17-62.fits): tried to move past end of file (107) : Moving to HDU 5
cfitsio error stack:
 {0}
lsst::afw::fits::FitsError: 'cfitsio error (rerun/calib/BIAS/2015-08-17/BIAS-2015-08-17-62.fits): tried to move past end of file (107) : Moving to HDU 5
cfitsio error stack:
'

ingestCalibs INFO: rerun/calib/BIAS/2015-08-17/BIAS-2015-08-17-62.fits --<copy>--> /sas3/projects/hydra/decam_test/CALIB/BIAS/2015-08-17/BIAS-2015-08-17-62.fits

In the end, the files each end up in the CALIB/BIAS directory where they should go, and have been registered in the calibRegistry database as well. However, when running the next step to generate the darks, I end up with this error message:

constructDark.py . --calib=CALIB/ --rerun=calib/ --job=darks --id visit=464646 --cores=64

4687 INFO  2021-02-02T13:06:14.933-0600 dark.isr: Performing ISR on sensor DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 46, 'ccdnum': 42, 'object': 'preflats-dark90s'}, tag=set()).
4688 INFO  2021-02-02T13:06:14.933-0600 dark.isr: Performing ISR on sensor DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 47, 'ccdnum': 43, 'object': 'preflats-dark90s'}, tag=set()).
4647 WARN  2021-02-02T13:06:15.322-0600 dark: Unable to process DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 35, 'ccdnum': 2, 'object': 'preflats-dark90s'}, tag=set()): Unable to retrieve cpBias for DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 35, 'ccdnum': 2, 'object': 'preflats-dark90s'}, tag=set()): No locations for get: datasetType:cpBias dataId:DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 35, 'ccdnum': 2, 'object': 'preflats-dark90s'}, tag=set()).
RuntimeError on anacreon:4647 in run: Unable to retrieve cpBias for DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 35, 'ccdnum': 2, 'object': 'preflats-dark90s'}, tag=set()): No locations for get: datasetType:cpBias dataId:DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 35, 'ccdnum': 2, 'object': 'preflats-dark90s'}, tag=set()).
Traceback (most recent call last):
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/21.0.0+ee58a624b3/python/lsst/ip/isr/isrTask.py", line 1728, in getIsrExposure
    exp = dataRef.get(datasetType, immediate=immediate)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/21.0.0+48431fa087/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
    return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/daf_persistence/21.0.0+48431fa087/python/lsst/daf/persistence/butler.py", line 1399, in get
    raise NoResults("No locations for get:", datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:cpBias dataId:DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 35, 'ccdnum': 2, 'object': 'preflats-dark90s'}, tag=set())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py", line 112, in wrapper
    return func(*args, **kwargs)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py", line 1069, in run
    while not menu[command]():
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py", line 239, in wrapper
    return func(*args, **kwargs)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py", line 1087, in reduce
    result = self._processQueue(context, func, [(index, data)], *args, **kwargs)[0]
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py", line 546, in _processQueue
    return self._reduceQueue(context, None, func, queue, *args, **kwargs)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py", line 572, in _reduceQueue
    resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py", line 572, in <listcomp>
    resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/21.0.0+c33e90402a/python/lsst/pipe/drivers/constructCalibs.py", line 625, in process
    exposure = self.processSingle(sensorRef, **kwargs)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/21.0.0+c33e90402a/python/lsst/pipe/drivers/constructCalibs.py", line 983, in processSingle
    exposure = CalibTask.processSingle(self, sensorRef)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/21.0.0+c33e90402a/python/lsst/pipe/drivers/constructCalibs.py", line 644, in processSingle
    return self.isr.runDataRef(dataRef).exposure
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_base/21.0.0+544a109665/python/lsst/pipe/base/timer.py", line 157, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/21.0.0+ee58a624b3/python/lsst/ip/isr/isrTask.py", line 1684, in runDataRef
    isrData = self.readIsrData(sensorRef, ccdExposure)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/21.0.0+ee58a624b3/python/lsst/ip/isr/isrTask.py", line 1065, in readIsrData
    if self.config.doBias else None)
  File "/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/21.0.0+ee58a624b3/python/lsst/ip/isr/isrTask.py", line 1731, in getIsrExposure
    raise RuntimeError("Unable to retrieve %s for %s: %s." % (datasetType, dataRef.dataId, exc1))
RuntimeError: Unable to retrieve cpBias for DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 35, 'ccdnum': 2, 'object': 'preflats-dark90s'}, tag=set()): No locations for get: datasetType:cpBias dataId:DataId(initialdata={'visit': 464646, 'date': '2015-08-17', 'filter': 'Y', 'hdu': 35, 'ccdnum': 2, 'object': 'preflats-dark90s'}, tag=set()).
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2

So in the end whatever has been ingested and registered doesn’t actually work for the next steps.

Reading through some of the other support tickets about DECam and obs_decam I suspect there’s some sort of clash between regular BIAS’es and cpBias’es, but I have no idea on how to fix this or work around it, so any pointers or advice would be awesome.

Thanks
Ralf

mrawls · February 2, 2021, 7:44pm

You did ingestion right - those errors are normal when you’re ingesting calibs that aren’t multi-extension FITS files.

EDIT TO ADD: You maybe didn’t quite do ingestion right, though you didn’t break it… typically you want to run ingestImagesDecam.py on the raw bias and flat frames and only run ingestCalibs.py on the constructed calibs. Although, now that I read it again, perhaps that’s exactly what you did, but only for biases, so far. Carry on.

You’re correct that it is looking for a cpBias but you have a bias. We don’t usually do darks for DECam, just biases and flats. When I run constructFlat.py in a very similar way, I add --config isr.doDark=False isr.biasDataProductName=bias. That should fix it. When you are ready to do image processing, you will also need to specify --config isr.flatDataProductName=flat. We should probably update these defaults sometime soon…