constructBias.py raise NoResults error

Support
,
#1

Hello,

I am running the detrending on HSC data following the HSC pipeline tutorial (http://hsc.mtk.nao.ac.jp/pipedoc_5_e/tutorial_e/index.html#) and I get stuck in the processing of the bias. I am running :

constructBias.py input --calib CALIB --rerun output --id field=BIAS --batch-type=smp --cores=1

And the error message I get is :

bias WARN: Unable to process DataId(initialdata={'field': 'BIAS', 'dateObs': '2014-11-28', 'pointing': 1062, 'filter': 'HSC-I', 'visit': 16382, 'ccd': 49, 'taiObs': '2014-11-28', 'expTime': 0.0}, tag=set()): No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={'field': 'BIAS', 'dateObs': '2014-11-28', 'pointing': 1062, 'filter': 'HSC-I', 'visit': 16382, 'ccd': 49, 'taiObs': '2014-11-28', 'expTime': 0.0}, tag=set())
NoResults on piecld00.isdc.unige.ch:3580 in reduce: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={'field': 'BIAS', 'dateObs': '2014-11-28', 'pointing': 1062, 'filter': 'HSC-I', 'visit': 16382, 'ccd': 49, 'taiObs': '2014-11-28', 'expTime': 0.0}, tag=set())
Traceback (most recent call last):
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/6.0b7/python/lsst/ctrl/pool/pool.py", line 113, in wrapper
    return func(*args, **kwargs)
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/6.0b7/python/lsst/ctrl/pool/pool.py", line 239, in wrapper
    return func(*args, **kwargs)
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/6.0b7/python/lsst/ctrl/pool/pool.py", line 715, in reduce
    *args, **kwargs)
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/6.0b7/python/lsst/ctrl/pool/pool.py", line 572, in _reduceQueue
    resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/6.0b7/python/lsst/ctrl/pool/pool.py", line 572, in <listcomp>
    resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b7/python/lsst/pipe/drivers/constructCalibs.py", line 619, in process
    exposure = self.processSingle(sensorRef, **kwargs)
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b7/python/lsst/pipe/drivers/constructCalibs.py", line 638, in processSingle
    return self.isr.runDataRef(dataRef).exposure
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_subaru/6.0b6-hsc+1/python/lsst/obs/subaru/isr.py", line 381, in runDataRef
    opticsTransmission = sensorRef.get("transmission_optics")
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/6.0b5-hsc+1/python/lsst/daf/persistence/butlerSubset.py", line 198, in get
    return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
  File "/opt/lsst/6.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/6.0b5-hsc+1/python/lsst/daf/persistence/butler.py", line 1410, in get
    raise NoResults("No locations for get:", datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={'field': 'BIAS', 'dateObs': '2014-11-28', 'pointing': 1062, 'filter': 'HSC-I', 'visit': 16382, 'ccd': 49, 'taiObs': '2014-11-28', 'expTime': 0.0}, tag=set())
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

I guess that something wrong happened somewhere before in the process or that I miss some data. I am using the version 6 of the HSC pipeline.

Any help, answer or discussion would be really appreciated.

Thanks in advance.

Guillaume

#2

I’m at a meeting a the moment, and I’m not super familiar with HSC data, but my guess is that the id argument you’re using to collect the data isn’t correct. I don’t know where it’s stored in the HSC headers, but it’s probably something more like IMGTYPE or OBSTYPE or something like that.

You can probably find out by taking a frame you know is a bias and looking at the header info, or looking at the translator functions in the obs_package to see how that info is being captured.

Also, @price might be more useful here.

#3

No, this is a failure to install the transmission curves. The clue is datasetType:transmission_optics in the error.

lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={'field': 'BIAS', 'dateObs': '2014-11-28', 'pointing': 1062, 'filter': 'HSC-I', 'visit': 16382, 'ccd': 49, 'taiObs': '2014-11-28', 'expTime': 0.0}, tag=set())
#4

Ah, sorry, good catch. I must admit that I just saw the good old no locations for get() error and assumed like a fool without scrutinizing :pensive:

#5

Thanks

It seems to work when i execute :

constructBias.py input --calib CALIB --rerun output --id field=BIAS --batch-type=smp --cores=1 -c isr.doAttachTransmissionCurve=False

Do I need to force not to use the Transmission Curves for all the detrending/processing ? Does this will have an implication later in the process ?

Thanks a lot for your answers !

#6

Apparently the process of the bias has been finished without raising another error, but there is no data written at the end in the CALIB directory. I checked the rerun directory and the intermediate files seem to have been created.

Does someone have an idea of the origin of the problem ?

Thanks a lot again !

#7

The calib construction scripts deliberately do not write the output in the CALIB directory. You need to use ingestCalibs.py to move the created master calibs into the calib registry.

For a (very verbose) log of the calib construction process, have a look at calib-20180117.txt.