Cosmic ray detection issue in dense fields

I’m running the characterizeImage task on data from DECam using version w.2022.06 of the pipelines. The images are of very dense fields in the galactic plane. I am finding that for some of the visits and detectors there are a very large number of cosmic rays being detected and repaired. Typically, this causes the pipeline to fail as the number of cosmic ray pixels exceeds the obs_decam default of 100,000:

Execution of task 'characterizeImage' on quantum {instrument: 'DECam', detector: 4, visit: 975920, ...} failed. Exception LengthError: 
  File "src/", line 724, in std::vector<std::shared_ptr<lsst::afw::detection::Footprint> > lsst::meas::algorithms::findCosmicRays(MaskedImageT&, const lsst::afw::detection::Psf&, double, const lsst::daf::base::PropertySet&, bool) [with MaskedImageT = lsst::afw::image::MaskedImage<float>]
    Too many CR pixels (max 100000) {0}
lsst::pex::exceptions::LengthError: 'Too many CR pixels (max 100000)'

I’ve tried increasing this number to get the processing to succeed, but find that I have to increase it so far as to make some of the data unusable or suspect: for example, 1,000,000 is ~12% of the detector.

I’m finding that this effect is variable with visit and detector number, which makes me think it is due to an observational effect. To explore this effect, I’ve plotted the same detector from 2 visits of the same field with cosmic ray masks overplotted:

Sorry if they are not very visible with color, size, and compression, here is a full resolution image:

What I find is that the cosmic rays seem to trace the background i.e. where the density of sources drops. I also find that the number of cosmic rays detected (9794 and 22488) are much, much higher than what I’d expect (no surprise given the error). For a less dense field and similar exposure time, the CR repair process finds 20-120 cosmic rays.

This effect is quite common, causing processing failures in ~18% of the detectors from visits to the dense field. So my solution is to turn off cosmic ray detection and repair when processing dense fields. But I am posting here in case anyone has insight about what causes this, potential fixes, and expected behavior when processing dense fields.

Section 3.1 of DMTN-171 discusses the issue of cosmic ray detection in crowded fields and proposes modifying configurables like the the detection thresholds for sources (for background measurement and PSF modelling) and cosmic rays. However it seems that the nCrPixelMax is set high.

The final component that required modification is the cosmic ray detection and repair algorithm. As noted above, the assumptions behind the pixel value statistics are incorrect in crowded fields. We set the detection thresholds to the same values as for source detection ( Table 3 ), and while this works in most cases, for just under 1% of the exposures fails with a fatal error. In these cases the failure appears to be due to every pixel in the image being identified as a cosmic ray. This failure suggests that our cosmic ray detection algorithm needs improvement and should be investigated further, but because of the low number of exposures affected we simply increased the number of pixels required to trigger the failure. This does not solve the problem, but it allows us to continue processing these exposures to make sure that there are no additional problems

Seems to concur with what I’ve reported, although with disparity in number of CCD failures, 1% vs my 18%. Although I haven’t changed or upped to 10,000,000 per that report, which may reduce the failure rate to 1%.