Scarcity of sources leads to failure of PSF determination and measureApCorr in characterizeImageTask

Support
#1

Hello. I am trying to run characterizeImageTask on data from DECam. I am finding that for a small fraction of visits and detectors (observing the same field), that the PSF determination and measureApCorr sub tasks are failing. There are three places I am seeing errors (from different visits/detectors):

  1. No PSF Stars because no sources are detected at 50 sigma:
lsst.characterizeImage.detection Detected 0 positive peaks in 0 footprints and 1 negative peaks in 1 footprints to 50 sigma
lsst.characterizeImage.detection Resubtracting the background after object detection
lsst.characterizeImage.measurement Measuring 0 sources (0 parents, 0 children) 
lsst.characterizeImage.measurePsf Measuring PSF
lsst.ctrl.mpexec.singleQuantumExecutor Execution of task 'characterizeImage' on quantum {instrument: 'DECam', detector: 27, visit: 978094, ...} failed. Exception RuntimeError: No objects passed our cuts for consideration as psf stars
  1. PSF determination fails with a strange error message when there are few sources to determine the PSF model (I have seen this happen with ~3 PSF sources as well):
lsst.characterizeImage.detection Detected 107 positive peaks in 36 footprints and 0 negative peaks in 0 footprints to 50 sigma
lsst.characterizeImage.detection Resubtracting the background after object detection
lsst.characterizeImage.measurement Measuring 36 sources (36 parents, 0 children) 
lsst.characterizeImage.measurePsf Measuring PSF
lsst.characterizeImage.measurePsf PSF star selector found 1 candidates
lsst.characterizeImage.measurePsf.reserve Reserved 0/1 sources
lsst.characterizeImage.measurePsf Sending 1 candidates to PSF determiner
lsst.characterizeImage.measurePsf.psfDeterminer NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
lsst.characterizeImage.measurePsf PSF determination using 1/1 stars.
lsst.ctrl.mpexec.singleQuantumExecutor Execution of task 'characterizeImage' on quantum {instrument: 'DECam', detector: 1, visit: 1002843, ...} failed. Exception InvalidParameterError: 
  File "src/PsfexPsf.cc", line 233, in virtual std::shared_ptr<lsst::afw::image::Image<double> > lsst::meas::extensions::psfex::PsfexPsf::_doComputeImage(const Point2D&, const lsst::afw::image::Color&, const Point2D&) const
    Only spatial variation (ndim == 2) is supported; saw 0 {0}
lsst::pex::exceptions::InvalidParameterError: 'Only spatial variation (ndim == 2) is supported; saw 0'
  1. PSF determination succeeds, but all sources are flagged and so aren’t selected by the measureApCorr source selector:
lsst.characterizeImage PSF estimation initialized with 'simple' PSF
lsst.characterizeImage.repair Identified 15 cosmic rays.
lsst.characterizeImage.detection Detected 54 positive peaks in 48 footprints and 0 negative peaks in 0 footprints to 50 sigma
lsst.characterizeImage.detection Resubtracting the background after object detection
lsst.characterizeImage.measurement Measuring 48 sources (48 parents, 0 children) 
lsst.characterizeImage.measurePsf Measuring PSF
lsst.characterizeImage.measurePsf PSF star selector found 26 candidates
lsst.characterizeImage.measurePsf.reserve Reserved 5/26 sources
lsst.characterizeImage.measurePsf Sending 21 candidates to PSF determiner
lsst.characterizeImage.measurePsf.psfDeterminer NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
lsst.characterizeImage.measurePsf PSF determination using 16/21 stars.
lsst.characterizeImage iter 2; PSF sigma=4.44, dimensions=(41, 41); median background=119.86
lsst.characterizeImage.repair Identified 16 cosmic rays.
lsst.characterizeImage.measurement Measuring 48 sources (48 parents, 0 children) 
lsst.characterizeImage.measureApCorr Measuring aperture corrections for 31 flux fields
lsst.characterizeImage.measureApCorr.sourceSelector Selected 0/48 sources
lsst.ctrl.mpexec.singleQuantumExecutor Execution of task 'characterizeImage' on quantum {instrument: 'DECam', detector: 35, visit: 987283, ...} failed. Exception RuntimeError: Unable to measure aperture correction for required algorithm 'base_GaussianFlux': only 0 sources, but require at least 2.

I have tried to lower the detection threshold from 50 sigma to 20, 30, and 40 sigma to see if I can recover the tasks that fail due to (1), (2), and (3), but it seems to not be a consistent fix across visits/detectors and it sometimes throws the issue downstream e.g. from (1) to (2).

I’ve seen that this happens across a variety of detectors and not always the same ones in sequential visits of the same field, so my guess is this is related to the weather/sky background varying so that fewer sources of lower quality are detected. I’m not too worried about this as this affects (as an initial guess) ~0.1% of detectors I’ve processed.

But I am throwing this out there in case anyone has thoughts on a fix.

#2

What version of the software are you using?

#3

I’m using weekly w.2022.06. I can upgrade if needed.

#4

Have you looked at the images that are failing to see if there’s anything obviously off about them?

#5

For one of the measureApCorr failures, it like the data quality got worse, perhaps due to a change in seeing or background. Here are plots of the raws, postISRCCDs, and icExps for adjacent visits and detectors for that case:

Full res: raw, postISRCCD, icExp

Visit 987283, detector 35 fails with:

lsst.characterizeImage INFO: PSF estimation initialized with 'simple' PSF
lsst.characterizeImage.repair INFO: Identified 17 cosmic rays.
lsst.characterizeImage.detection INFO: Detected 54 positive peaks in 39 footprints and 0 negative peaks in 0 footprints to 50 sigma
lsst.characterizeImage.detection INFO: Resubtracting the background after object detection
lsst.characterizeImage.measurement INFO: Measuring 39 sources (39 parents, 0 children) 
lsst.characterizeImage.measurePsf INFO: Measuring PSF
lsst.characterizeImage.measurePsf INFO: PSF star selector found 9 candidates
lsst.characterizeImage.measurePsf.reserve INFO: Reserved 2/9 sources
lsst.characterizeImage.measurePsf INFO: Sending 7 candidates to PSF determiner
lsst.characterizeImage.measurePsf.psfDeterminer WARNING: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value

> WARNING: 1st context group-degree lowered (not enough samples)

lsst.characterizeImage.measurePsf INFO: PSF determination using 7/7 stars.
lsst.characterizeImage INFO: iter 1; PSF sigma=4.45, dimensions=(41, 41); median background=119.87
lsst.characterizeImage INFO: PSF estimation initialized with 'simple' PSF
lsst.characterizeImage.repair INFO: Identified 15 cosmic rays.
lsst.characterizeImage.detection INFO: Detected 54 positive peaks in 48 footprints and 0 negative peaks in 0 footprints to 50 sigma
lsst.characterizeImage.detection INFO: Resubtracting the background after object detection
lsst.characterizeImage.measurement INFO: Measuring 48 sources (48 parents, 0 children) 
lsst.characterizeImage.measurePsf INFO: Measuring PSF
lsst.characterizeImage.measurePsf INFO: PSF star selector found 26 candidates
lsst.characterizeImage.measurePsf.reserve INFO: Reserved 5/26 sources
lsst.characterizeImage.measurePsf INFO: Sending 21 candidates to PSF determiner
lsst.characterizeImage.measurePsf.psfDeterminer WARNING: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
lsst.characterizeImage.measurePsf INFO: PSF determination using 16/21 stars.
lsst.characterizeImage INFO: iter 2; PSF sigma=4.44, dimensions=(41, 41); median background=119.86
lsst.characterizeImage.repair INFO: Identified 16 cosmic rays.
lsst.characterizeImage.measurement INFO: Measuring 48 sources (48 parents, 0 children) 
lsst.characterizeImage.measureApCorr INFO: Measuring aperture corrections for 31 flux fields
lsst.characterizeImage.measureApCorr.sourceSelector INFO: Selected 0/48 sources
lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'characterizeImage' on quantum {instrument: 'DECam', detector: 35, visit: 987283, ...} failed. Exception RuntimeError: Unable to measure aperture correction for required algorithm 'base_GaussianFlux': only 0 sources, but require at least 2.

in comparison to visit 987284, detector 35:

lsst.characterizeImage INFO: PSF estimation initialized with 'simple' PSF
lsst.characterizeImage.repair INFO: Identified 88 cosmic rays.
lsst.characterizeImage.detection INFO: Detected 136 positive peaks in 68 footprints and 0 negative peaks in 0 footprints to 50 sigma
lsst.characterizeImage.detection INFO: Resubtracting the background after object detection
lsst.characterizeImage.measurement INFO: Measuring 68 sources (68 parents, 0 children) 
lsst.characterizeImage.measurePsf INFO: Measuring PSF
lsst.characterizeImage.measurePsf INFO: PSF star selector found 25 candidates
lsst.characterizeImage.measurePsf.reserve INFO: Reserved 5/25 sources
lsst.characterizeImage.measurePsf INFO: Sending 20 candidates to PSF determiner
lsst.characterizeImage.measurePsf.psfDeterminer WARNING: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
lsst.characterizeImage.measurePsf INFO: PSF determination using 16/20 stars.
lsst.characterizeImage INFO: iter 1; PSF sigma=5.22, dimensions=(41, 41); median background=712.93
lsst.characterizeImage INFO: PSF estimation initialized with 'simple' PSF
lsst.characterizeImage.repair INFO: Identified 75 cosmic rays.
lsst.characterizeImage.detection INFO: Detected 77 positive peaks in 57 footprints and 0 negative peaks in 0 footprints to 50 sigma
lsst.characterizeImage.detection INFO: Resubtracting the background after object detection
lsst.characterizeImage.measurement INFO: Measuring 57 sources (57 parents, 0 children) 
lsst.characterizeImage.measurePsf INFO: Measuring PSF
lsst.characterizeImage.measurePsf INFO: PSF star selector found 28 candidates
lsst.characterizeImage.measurePsf.reserve INFO: Reserved 6/28 sources
lsst.characterizeImage.measurePsf INFO: Sending 22 candidates to PSF determiner
lsst.characterizeImage.measurePsf.psfDeterminer WARNING: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
lsst.characterizeImage.measurePsf INFO: PSF determination using 17/22 stars.
lsst.characterizeImage INFO: iter 2; PSF sigma=5.22, dimensions=(41, 41); median background=712.87
lsst.characterizeImage.repair INFO: Identified 75 cosmic rays.
lsst.characterizeImage.measurement INFO: Measuring 57 sources (57 parents, 0 children) 
lsst.characterizeImage.measureApCorr INFO: Measuring aperture corrections for 31 flux fields
lsst.characterizeImage.measureApCorr.sourceSelector INFO: Selected 8/57 sources
lsst.characterizeImage.measureApCorr INFO: Aperture correction for base_GaussianFlux: RMS 0.001297 from 8
lsst.characterizeImage.measureApCorr INFO: Aperture correction for base_PsfFlux: RMS 0.000281 from 8
...

Interestingly, I find that for one of “no suitable PSF images”, the ISR step masks out one of the amplifiers.

Full res: raw, postISRCCD, icExp

Visit 978094, detector 27 fails with:

lsst.characterizeImage INFO: PSF estimation initialized with 'simple' PSF
lsst.characterizeImage.repair INFO: Identified 1 cosmic rays.
lsst.characterizeImage.detection INFO: Detected 0 positive peaks in 0 footprints and 1 negative peaks in 1 footprints to 50 sigma
lsst.characterizeImage.detection INFO: Resubtracting the background after object detection
lsst.characterizeImage.measurement INFO: Measuring 0 sources (0 parents, 0 children) 
lsst.characterizeImage.measurePsf INFO: Measuring PSF
lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'characterizeImage' on quantum {instrument: 'DECam', detector: 27, visit: 978094, ...} failed. Exception RuntimeError: No objects passed our cuts for consideration as psf stars

in comparison to visit 978093, detector 27:

lsst.characterizeImage INFO: PSF estimation initialized with 'simple' PSF
lsst.characterizeImage.repair INFO: Identified 162 cosmic rays.
lsst.characterizeImage.detection INFO: Detected 176 positive peaks in 71 footprints and 0 negative peaks in 0 footprints to 50 sigma
lsst.characterizeImage.detection INFO: Resubtracting the background after object detection
lsst.characterizeImage.measurement INFO: Measuring 71 sources (71 parents, 0 children) 
lsst.characterizeImage.measurePsf INFO: Measuring PSF
lsst.characterizeImage.measurePsf INFO: PSF star selector found 50 candidates
lsst.characterizeImage.measurePsf.reserve INFO: Reserved 10/50 sources
lsst.characterizeImage.measurePsf INFO: Sending 40 candidates to PSF determiner
lsst.characterizeImage.measurePsf.psfDeterminer WARNING: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
lsst.characterizeImage.measurePsf INFO: PSF determination using 39/40 stars.
lsst.characterizeImage INFO: iter 1; PSF sigma=2.62, dimensions=(41, 41); median background=8744.28
lsst.characterizeImage INFO: PSF estimation initialized with 'simple' PSF
lsst.characterizeImage.repair INFO: Identified 194 cosmic rays.
lsst.characterizeImage.detection INFO: Detected 124 positive peaks in 76 footprints and 0 negative peaks in 0 footprints to 50 sigma
lsst.characterizeImage.detection INFO: Resubtracting the background after object detection
lsst.characterizeImage.measurement INFO: Measuring 76 sources (76 parents, 0 children) 
lsst.characterizeImage.measurePsf INFO: Measuring PSF
lsst.characterizeImage.measurePsf INFO: PSF star selector found 53 candidates
lsst.characterizeImage.measurePsf.reserve INFO: Reserved 11/53 sources
lsst.characterizeImage.measurePsf INFO: Sending 42 candidates to PSF determiner
lsst.characterizeImage.measurePsf.psfDeterminer WARNING: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
lsst.characterizeImage.measurePsf INFO: PSF determination using 42/42 stars.
lsst.characterizeImage INFO: iter 2; PSF sigma=2.60, dimensions=(41, 41); median background=8744.23
lsst.characterizeImage.repair INFO: Identified 191 cosmic rays.
lsst.characterizeImage.measurement INFO: Measuring 76 sources (76 parents, 0 children) 
lsst.characterizeImage.measureApCorr INFO: Measuring aperture corrections for 31 flux fields
lsst.characterizeImage.measureApCorr.sourceSelector INFO: Selected 21/76 sources
lsst.characterizeImage.measureApCorr INFO: Aperture correction for base_GaussianFlux: RMS 0.003507 from 20
lsst.characterizeImage.measureApCorr INFO: Aperture correction for base_PsfFlux: RMS 0.006995 from 21
...

For the ISR step removing one amplifier, what I see from the logs is that the estimated sky level is much higher in the failing detector:

  • failing {instrument: 'DECam', detector: 27, exposure: 978094, ...} 
lsst.isrForCrosstalkSources INFO: Measuring background level.
lsst.isrForCrosstalkSources INFO: Flattened sky level: 30797.916016 +/- 306.558563.
lsst.isrForCrosstalkSources INFO: Measuring sky levels in 8x16 grids: 30765.426219.
lsst.isrForCrosstalkSources INFO: Sky flatness in 8x16 grids - pp: 1.010008 rms: 0.411846.

lsst.isr INFO: Measuring background level.
lsst.isr INFO: Flattened sky level: 27939.449219 +/- 97.188000.
lsst.isr INFO: Measuring sky levels in 8x16 grids: 27940.806944.
lsst.isr INFO: Sky flatness in 8x16 grids - pp: 0.008280 rms: 0.001806.
  • passing {instrument: 'DECam', detector: 27, exposure: 978093, ...}:
lsst.isrForCrosstalkSources INFO: Measuring background level.
lsst.isrForCrosstalkSources INFO: Flattened sky level: 9023.625000 +/- 281.248587.
lsst.isrForCrosstalkSources INFO: Measuring sky levels in 8x16 grids: 9052.430920.
lsst.isrForCrosstalkSources INFO: Sky flatness in 8x16 grids - pp: 0.362456 rms: 0.039461.

lsst.isr INFO: Measuring background level.
lsst.isr INFO: Flattened sky level: 8618.151367 +/- 95.992212.
lsst.isr INFO: Measuring sky levels in 8x16 grids: 8623.766672.
lsst.isr INFO: Sky flatness in 8x16 grids - pp: 0.039558 rms: 0.009853.
#6

These visits are different bands and have different exposure times, so a change in background level is expected. detector 27, exposure 978094 (failing) is 130 sec in i-band a.nd detector 27, exposure 978093 (passing) is 86 sec in r-band.