imageDifference -- Cannot find any objects suitable for KernelCandidacy

Support
#1

I’m running into an issue trying to use imageDifference.py where for a certain dataset (117 nearly coaligned images of a single HSC chip) the script crashes with the error “Cannot find any objects suitable for KernelCandidacy”. I’ve copied the full output below.

oddly this works perfectly fine with most other chips in the same dataset. Just not for a few of the edge chips. This error only seems to happen with this particular dataset.

This happens regardless of the skymap shape (single patch, multi patch, patches that include other chips, etc.).

I can share the images if that’s useful for debugging.

The script output:

(lsst-scipipe-4d7b902) > imageDifference.py /arc/projects/NewHorizons/HSC_21_June-lsst/ @03447/imageDifference/inputs/0255552_4.txt
CameraMapper INFO: Loading exposure registry from /arc/projects/NewHorizons/HSC_21_June-lsst/registry.sqlite3
CameraMapper INFO: Loading calib registry from /arc/projects/NewHorizons/HSC_21_June-lsst/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /arc/projects/NewHorizons/HSC_21_June-lsst/CALIB/calibRegistry.sqlite3
HscMapper WARN: Unable to find calib root directory
HscMapper WARN: Unable to find calib root directory
HscMapper WARN: Unable to find calib root directory
HscMapper WARN: Unable to find calib root directory
HscMapper WARN: Unable to find calib root directory
HscMapper WARN: Unable to find calib root directory
HscMapper WARN: Unable to find calib root directory
CameraMapper INFO: Loading calib registry from /arc/projects/NewHorizons/HSC_21_June-lsst/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading exposure registry from /arc/projects/NewHorizons/HSC_21_June-lsst/registry.sqlite3
CameraMapper INFO: Loading calib registry from /arc/projects/NewHorizons/HSC_21_June-lsst/CALIB/calibRegistry.sqlite3
root INFO: Running: /opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/bin/imageDifference.py /arc/projects/NewHorizons/HSC_21_June-lsst/ @03447/imageDifference/inputs/0255552_4.txt
WARNING: You are using OpenBLAS with multiple threads (32), but have not
specified the number of threads using one of the OpenBLAS environment variables:
OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
This may indicate that you are unintentionally using multiple threads, which may
cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
what you are doing and want threads enabled implicitly, set the environment
variable LSST_ALLOW_IMPLICIT_THREADS.
imageDifference INFO: Processing {‘visit’: 255552, ‘ccd’: 4, ‘pointing’: 3447, ‘filter’: ‘HSC-R2’, ‘field’: ‘NHF210609’, ‘dateObs’: ‘2021-06-09’, ‘taiObs’: ‘2021-06-09’, ‘expTime’: 90.0}
imageDifference.getTemplate INFO: Using skyMap tract 0
imageDifference.getTemplate INFO: Assembling 1 coadd patches
imageDifference.getTemplate INFO: exposure dimensions=(2048, 4176); coadd dimensions=(4014, 2081)
imageDifference.getTemplate INFO: Reading patch {‘datasetType’: ‘deepCoadd_sub’, ‘bbox’: Box2I(minimum=Point2I(0, 6160), dimensions=Extent2I(3030, 1840)), ‘tract’: 0, ‘patch’: ‘0,1’, ‘numSubfilters’: 3}
imageDifference INFO: Subtracting images
imageDifference.subtract INFO: Template Wcs : 5.047406,-0.379212 → 5.043886,-0.377521
imageDifference.subtract INFO: Science Wcs : 5.043897,-0.379208 → 5.047395,-0.377526
imageDifference.subtract INFO: Astrometrically registering template to science image
imageDifference.subtract INFO: templateFwhmPix: 4.792198406197638
imageDifference.subtract INFO: scienceFwhmPix: 5.510377698803702
ip.diffim.generateAlardLuptonBasisList INFO: Reference psf fwhm is the greater, normal convolution mode
imageDifference.subtract.selectDetection INFO: Detected 5477 positive peaks in 1603 footprints to 10 sigma
imageDifference.subtract.selectMeasurement INFO: Measuring 1603 sources (1603 parents, 0 children)
imageDifference.subtract INFO: Growing 1603 kernel candidate stars by 27 pixels
imageDifference.subtract INFO: Selected 0 / 1603 sources for KernelCandidacy
imageDifference FATAL: Failed on dataId={‘visit’: 255552, ‘ccd’: 4, ‘pointing’: 3447, ‘filter’: ‘HSC-R2’, ‘field’: ‘NHF210609’, ‘dateObs’: ‘2021-06-09’, ‘taiObs’: ‘2021-06-09’, ‘expTime’: 90.0}: RuntimeError: Cannot find any objects suitable for KernelCandidacy
Traceback (most recent call last):
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/cmdLineTask.py”, line 388, in call
result = self.runTask(task, dataRef, kwargs)
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/cmdLineTask.py”, line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/imageDifference.py”, line 578, in runDataRef
doWarping=not self.config.doUseRegister
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_diffim/19.0.0+2/python/lsst/ip/diffim/imagePsfMatch.py”, line 635, in subtractExposures
convolveTemplate=convolveTemplate
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_diffim/19.0.0+2/python/lsst/ip/diffim/imagePsfMatch.py”, line 438, in matchExposures
templateExposure, scienceExposure, kernelSize, candidateList)
File “/opt/lsst/software/stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_diffim/19.0.0+2/python/lsst/ip/diffim/imagePsfMatch.py”, line 860, in makeCandidateList
raise RuntimeError(“Cannot find any objects suitable for KernelCandidacy”)
RuntimeError: Cannot find any objects suitable for KernelCandidacy

The contents of 03447/imageDifference/inputs/0255552_4.txt:

–rerun 03447_nowarpCompare_004:diff_noWarpCompare_004
–id visit=0255552 ccd=4
–clobber-config --configfile /arc/projects/NewHorizons/configs/imageDifference_config.py

The contents of imageDifference_config.py:

config.doSelectSources=False
config.kernelSourcesFromRef=True
config.doMatchSources=False
config.doMeasurement=False
config.doDetection=False
config.doDipoleFitting=False
config.doForcedMeasurement=False
config.doMerge=False
config.doWriteSources=False

#2

Hi Wes, just a quick note to say that I see this, but I’m not sure how to fix and so am circulating this issue with DIA experts.

#3

Hi again, there was a suggestion that the error message might indicate that the reference catalog may not overlap all areas, to the edges. Maybe you already checked that though?

#4

Thanks for responding @MelissaGraham

Yeah I had already checked that. Also checked many other obvious things including that the template patches fully overlapped the CORR imagery, and things like that. The simply failed in this way for ~30% of the chips in an HSC field. Odd. Where it did succeed, large swaths of images had very odd subtraction artifacts.

The solution was to treat each chip individually (single patch skymap centered on the chip, templates and warps only for that chip, etc.). Contrast that to our normal approach to just include all imagery from a field in the template+diff process, which has worked flawlessly for most other nights/fields in our program. The newer approach is perfectly fine for us because we have repeated imagery of the same field with zero intentional dithering, and so the solution is good.

Just very strange that the full frame version fell over.

#5

Hi Wes, thanks for responding with the solution that you did find, as it might help someone else who arrives here with the same issue.

Any further comments from anyone about the potential underlying cause – especially in light of the solution – would still be a helpful addition to this thread.