Cannot do fringe calibration for z-band DECam image processing

Support
, ,
#1

Hello. I am trying to process z-band images from DECam with the LSST pipelines. For the ISR step, the z- and Y-band images require fringe correction, however the pipelines fail to execute the ISR task on z-band images because I do not have fringe calibrations in my repository:

lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'isr' on quantum {instrument: 'DECam', detector: 5, exposure: 1045101, ...} failed. Exception KeyError: 'fringes'
Process task-{instrument: 'DECam', detector: 5, exposure: 1045101, ...}:
Traceback (most recent call last):
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+a8f678a356/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 131, in _executeJob
    quantumExecutor.execute(taskDef, quantum, butler)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+a8f678a356/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 194, in execute
    self.runQuantum(runTask, quantum, taskDef, butler)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+a8f678a356/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 550, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ip_isr/g9cb75138f3+51aefb3276/python/lsst/ip/isr/isrTask.py", line 1081, in runQuantum
    inputs['fringes'] = self.fringe.loadFringes(inputs['fringes'],
KeyError: 'fringes'

I am running the pipelines with DECam/calib included as an input collection, which is where curated calibrations have been written to.

I haven’t been able to find any fringe calibrations in the NOIRLab archive to make my own fringe template, so I am wondering if there is a fringe template compatible with the LSST pipelines that I can access and add to my repository. Looking around, I find that NOIRLab has released public fringe correction templates that are used in the community pipeline.

Thanks for any help with this.

#2

Fringe calibrations are generated from science images, so you should be able to create one from your own data (instructions on how to do so available here: Constructing calibrations — LSST Science Pipelines). The fringes are built by aggressively masking pixels that fall on sources, so if you have too few input exposures (or a very dense field), you may need to supplement with other science exposures to ensure the fringe signal is well sampled everywhere.
As for using the community pipeline fringes, there isn’t really a good way to ingest arbitrary calibrations right now. It is possible to manually construct the export.yaml that butler import uses, but that’s likely more challenging than simply constructing the fringe yourself.

#3

It should be possible to use butler ingest-files if you knew the dataset type and dataId that you needed to specify. Then you can certify them into the relevant calibration collection.

#4

Thanks, Christopher, I’ll see if I can make my own fringe calibrations.

On my first try using the docs you linked, I’m getting an error:

lsst.pipe.base.pipeTools.PipelineDataCycleError: Pipeline has data cycles:
   {'cpFringeIsr'} -> cpFringe -> {'cpFringeProc'}
   {'cpFringeProc'} -> cpFringeCombine -> {'fringe'}
   {'fringe'} -> isr -> {'cpFringeIsr'}

I don’t understand this, since I have doFringe: false in my pipeline definition file, so IsrTask shouldn’t depend on having fringe datasets. Do you know what could be happening?

#5

Also, do you have any sense for how bad it would be to construct a fringe template using very dense fields with ~270 exposures of the same target? I guess unless the observations of the target were dithered, the same spots on the detectors will get masked when producing the template because the sources will appear in the same place, so having many more exposures might not produce better sampling of the fringe pattern in that case.

#6

The first seems to indicate a bug somewhere. If you edit the pipeline definition file such that the connections on the isr task look like the following:

tasks:
  isr:
    class: lsst.ip.isr.IsrTask
    config:
      connections.ccdExposure: 'raw'
      connections.outputExposure: 'cpFringeIsr'
      connections.fringes: "noFringesAllowed"
      doBias: true
[...]

then you should be able to run the processing.

I would think that 270 exposures would be more than enough, although you may run into memory issues during the combination. An update that resolves that is in review, and should be available next week.