Error when creating calibration master files

Support
#1

Hello,

I’m trying to process calibration images using the gen2 stack.
I can successfully ingest all images with ingestImages.py
The files are all linked using the directory structure that I set in the policy file,
so I think that part is working well.

But when I try to construct master calibration frames with

constructDark.py DATA --rerun darkcombine --id dataType=dark --job dark --cores 1 --batch-type=smp

I get this error:

$ constructDark.py DATA --rerun darkcombine --id dataType=dark --job dark --cores 1 --batch-type=smp
root INFO: Loading config overrride file '/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_ctmo/v1/config/dark.py'
CameraMapper INFO: Loading exposure registry from /home/lsst/pipe/DATA/registry.sqlite3
root INFO: Loading config overrride file '/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_ctmo/v1/config/dark.py'
CameraMapper INFO: Loading exposure registry from /home/lsst/pipe/DATA/registry.sqlite3
root INFO: Running: -c DATA --rerun darkcombine --id dataType=dark --noExit
dark FATAL: Failed: Unable to determine output filename "dark_filename" from {'ccd': 1, 'filter': 'NONE', 'calibDate': '2020-10-11'}: No unique lookup for {'taiObs'} from DataId(initialdata={'ccd': 1, 'filter': 'NONE', 'calibDate': '2020-10-11'}, tag=set()): 0 matches
root ERROR: 1 dataRefs failed; not exiting as --noExit was set
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_drivers/20.0.0/python/lsst/pipe/drivers/constructCalibs.py", line 453, in runDataRef
    butler.get(self.calibName + "_filename", dataId)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/daf_persistence/20.0.0/python/lsst/daf/persistence/butler.py", line 1397, in get
    location = self._locate(datasetType, dataId, write=False)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/daf_persistence/20.0.0/python/lsst/daf/persistence/butler.py", line 1316, in _locate
    location = repoData.repo.map(datasetType, dataId, write=write)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/daf_persistence/20.0.0/python/lsst/daf/persistence/repository.py", line 239, in map
    loc = self._mapper.map(*args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/daf_persistence/20.0.0/python/lsst/daf/persistence/mapper.py", line 163, in map
    return func(self.validate(dataId), write)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_base/20.0.0/python/lsst/obs/base/cameraMapper.py", line 383, in mapClosure
    return mapping.map(mapper, dataId, write)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_base/20.0.0/python/lsst/obs/base/mapping.py", line 454, in map
    location = Mapping.map(self, mapper, dataId, write=write)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_base/20.0.0/python/lsst/obs/base/mapping.py", line 152, in map
    actualId = self.need(iter(self.keyDict.keys()), dataId)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_base/20.0.0/python/lsst/obs/base/mapping.py", line 316, in need
    lookups = self.lookup(newProps, newId)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/obs_base/20.0.0/python/lsst/obs/base/mapping.py", line 506, in lookup
    (columns, dataId, len(lookups)))
RuntimeError: No unique lookup for {'taiObs'} from DataId(initialdata={'ccd': 1, 'filter': 'NONE', 'calibDate': '2020-10-11'}, tag=set()): 0 matches

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_drivers/20.0.0/python/lsst/pipe/drivers/constructCalibs.py", line 365, in __call__
    result = task.runDataRef(**args)
  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-1a1d771/Linux64/pipe_drivers/20.0.0/python/lsst/pipe/drivers/constructCalibs.py", line 457, in runDataRef
    (self.calibName, dataId, e))
RuntimeError: Unable to determine output filename "dark_filename" from {'ccd': 1, 'filter': 'NONE', 'calibDate': '2020-10-11'}: No unique lookup for {'taiObs'} from DataId(initialdata={'ccd': 1, 'filter': 'NONE', 'calibDate': '2020-10-11'}, tag=set()): 0 matches

I think that the “taiObs” refers to this taiObs in yaml configuration in the ObsMapper policy file

  dark:
    columns:
      - ccd
      - taiObs

taiObs is a column in both the raw and raw_visit tables of registry and it’s populated with a date.
So, I don’t know what could be wrong, why the stack is not picking taiObs from registry or somewhere else.

Anyone can give me a hand on this one? Much appreciated.

Thanks.
M.

#2

I haven’t built darks, but I have built biases and flats. The error stating “No unique lookup” suggests to me that you may have ingested more than one raw dark with the same taiObs value.

#3

Oh, that could very well be. My taiObs are just the date with no time stamp. I’ll try adding the whole datetime info to taiObs and see if that fixes it. Thanks!

#4

Can you point us to the source for your obs package?
In particular, what is the template value for your dark dataset?

#5

Yes, sure. This is where we keep our obs_ctmo:

Template value for dark dataset is

template: CALIB/DARK/DARK_%(run)s_%(calibDate)s.fits

#6

What is run? constructDark.py doesn’t know about it. Where is it supposed to get the value?

#7

run is the RUN-ID in header file. It’s supposed to be an unique ID for every exposure. I copied that from obs_goto, it was frameId before.

config.register.unique = ["run", "ccd", "filter"]

I checked and it’s a column of raw table in the registry.

#8

So what should the run value of a master dark be?

#9

That’s a good question, I don’t think it needs one? I thought those were just for the raw exposures, and it wouldn’t translate to any other product.

#10

Agreed: I don’t think it makes sense to have run in the dark's template.

What else do you need in the template? You already have calibDate, which will allow you to have different darks for different dates. If you have multiple CCDs in the camera then you should include ccd. That’s probably all you need for a dark. For calibs involving light going through the optical system (e.g., flat, fringe, sky), you’ll probably also want filter.

#11

I’m confused, I think the template is for the raw darks ingested. All raw exposures, darks included, are ingested well by butler, as far as I could check. It’s only when I want to combine them into a master dark that I get the error.

It will complain about taiObs key not being on this list {'ccd': 1, 'filter': 'NONE', 'calibDate': '2020-10-11'}.

But I’m not sure why is taiObs even required?

#12

The dark dataset is for master darks (i.e., the output of constructDark.py), so its template should not include run (since it is the combination of multiple runs).

The error about taiObs is coming from the butler attempting to find the run value when it doesn’t have the information it needs. If you fix the template for dark (and any other similar calibs) so it only includes the available keys (which list is probably just calibDate, ccd and filter), you shouldn’t hit it any more.

1 Like
#13

Oh! I see now. That clarifies my confusion, thank you.