Correct use of imageDifference.py

fraserw · January 28, 2020, 11:47pm

Hi Community.

Trying to get imageDifference.py to subtract a merged template from the warps used to make said template.

The command I am trying to execute is (full output copied below):

imageDifference.py DATA_CFHT --rerun coadd:deepCoadd --id filter=r2 tract=0 patch=0,0 extension=1 visit=2426913 --templateId filter=r2 tract=0 patch=0,0 --clobber-config --config doSelectSources=False kernelSourcesFromRef=True

which crashes out eventually with the error:

imageDifference FATAL: Failed on dataId={‘filter’: ‘r2’, ‘tract’: ‘0’, ‘patch’: ‘0,0’, ‘extension’: 1, ‘visit’: 2426913, ‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘ccd’: 0, ‘state’: ‘p’, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}: RuntimeError: Template is not defined for the deepDiff_diaSrc dataset type, it must be set before it can be used.

Images are from Megacam, and processed successfully with processCcd, makeDiscreteSkyMap, makeCoaddTEmpExp, and assembleCoadds. All processed images and warps are in their original butler locations.

Probably just a mistake with how I am calling imageDifference.

Anyone know the solution?

Thanks

Full imageDifference output:

CameraMapper INFO: Loading exposure registry from /media/LSSTproc/DATA_CFHT/registry.sqlite3

root WARN: Unexpected ID tract; guessing type is “str”

root WARN: Unexpected ID patch; guessing type is “str”

root INFO: Running: /home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/bin/imageDifference.py DATA_CFHT --rerun coadd:deepCoadd --id filter=r2 tract=0 patch=0,0 extension=1 visit=2426913 --templateId filter=r2 tract=0 patch=0,0 --clobber-config --config doSelectSources=False kernelSourcesFromRef=True

WARNING: You are using OpenBLAS with multiple threads (4), but have not

specified the number of threads using one of the OpenBLAS environment variables:

OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.

This may indicate that you are unintentionally using multiple threads, which may

cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know

what you are doing and want threads enabled implicitly, set the environment

variable LSST_ALLOW_IMPLICIT_THREADS.

imageDifference INFO: Processing {‘filter’: ‘r2’, ‘tract’: ‘0’, ‘patch’: ‘0,0’, ‘extension’: 1, ‘visit’: 2426913, ‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘ccd’: 0, ‘state’: ‘p’, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}

imageDifference.getTemplate INFO: Using skyMap tract 0

imageDifference.getTemplate INFO: Assembling 2 coadd patches

imageDifference.getTemplate INFO: exposure dimensions=(2048, 4612); coadd dimensions=(1188, 2568)

imageDifference.getTemplate INFO: Reading patch {‘datasetType’: ‘deepCoadd_sub’, ‘bbox’: Box2I(minimum=Point2I(2330, 10914), dimensions=Extent2I(1188, 1186)), ‘tract’: 0, ‘patch’: ‘0,2’, ‘numSubfilters’: 3}

imageDifference.getTemplate INFO: Reading patch {‘datasetType’: ‘deepCoadd_sub’, ‘bbox’: Box2I(minimum=Point2I(2330, 11900), dimensions=Extent2I(1188, 1582)), ‘tract’: 0, ‘patch’: ‘0,3’, ‘numSubfilters’: 3}

imageDifference INFO: Subtracting images

imageDifference.subtract INFO: Template Wcs : 5.066344,-0.378546 -> 5.064267,-0.374407

imageDifference.subtract INFO: Science Wcs : 5.064279,-0.374418 -> 5.066335,-0.378531

imageDifference.subtract INFO: Astrometrically registering template to science image

imageDifference.subtract INFO: templateFwhmPix: 3.8609133057836065

imageDifference.subtract INFO: scienceFwhmPix: 4.421904801366045

ip.diffim.generateAlardLuptonBasisList INFO: Reference psf fwhm is the greater, normal convolution mode

imageDifference.subtract.selectDetection INFO: Detected 7661 positive peaks in 2662 footprints to 10 sigma

imageDifference.subtract.selectMeasurement INFO: Measuring 2662 sources (2662 parents, 0 children)

imageDifference.subtract INFO: Growing 2662 kernel candidate stars by 21 pixels

imageDifference.subtract INFO: Selected 2219 / 2662 sources for KernelCandidacy

imageDifference.subtract INFO: Matching Psf FWHM 3.86 -> 4.42 pix

ip.diffim.generateAlardLuptonBasisList INFO: Reference psf fwhm is the greater, normal convolution mode

imageDifference.subtract INFO: Final spatial kernel sum 228.711

imageDifference.subtract INFO: Spatial model condition number 4.283e+07

imageDifference.subtract INFO: Doing stats of kernel candidates used in the spatial fit.

imageDifference.subtract INFO: 2219 candidates total, 826 rejected, 1100 used

imageDifference.subtract INFO: Spatial kernel model well constrained; 1100 candidates, 3 terms, 27 bases

imageDifference.subtract INFO: Spatial background model appears well constrained; 1100 candidates, 6 terms

imageDifference INFO: Computing diffim PSF

imageDifference INFO: Running diaSource detection

imageDifference.detection INFO: Detected 5461 positive peaks in 1899 footprints and 5727 negative peaks in 2187 footprints to 5.5 sigma

imageDifference INFO: Merging detections into 1658 sources

imageDifference INFO: Running diaSource measurement: newDipoleFitting=True

imageDifference.measurement INFO: Measuring 1658 sources (1658 parents, 0 children)

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545015: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545016: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545017: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545021: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545048: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545094: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545167: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545187: edge failure

/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_diffim/19.0.0+2/python/lsst/ip/diffim/dipoleFitTask.py:871: RuntimeWarning: divide by zero encountered in double_scalars

signalToNoise = np.sqrt((fluxVal/fluxVar)**2 + (fluxValNeg/fluxVarNeg)**2)

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545305: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545374: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545431: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545464: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545511: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545541: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545551: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545596: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545649: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545654: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545772: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748545881: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748545908: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546012: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546099: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546145: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546153: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546236: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546270: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546324: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546331: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546333: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546393: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546397: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546419: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546444: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin not run on record 375246430748546450: edge failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546538: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546577: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546588: dipole fit failure

/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_diffim/19.0.0+2/python/lsst/ip/diffim/dipoleFitTask.py:871: RuntimeWarning: divide by zero encountered in double_scalars

signalToNoise = np.sqrt((fluxVal/fluxVar)**2 + (fluxValNeg/fluxVarNeg)**2)

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546629: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546637: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546641: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546648: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546652: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546659: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546662: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546665: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546667: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546668: dipole fit failure

ip_diffim_DipoleFit WARN: DipoleFitPlugin failed on record 375246430748546671: dipole fit failure

imageDifference.forcedMeasurement INFO: Performing forced measurement on 1658 sources

imageDifference FATAL: Failed on dataId={‘filter’: ‘r2’, ‘tract’: ‘0’, ‘patch’: ‘0,0’, ‘extension’: 1, ‘visit’: 2426913, ‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘ccd’: 0, ‘state’: ‘p’, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}: RuntimeError: Template is not defined for the deepDiff_diaSrc dataset type, it must be set before it can be used.

Traceback (most recent call last):

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/cmdLineTask.py”, line 388, in call

result = self.runTask(task, dataRef, kwargs)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/cmdLineTask.py”, line 447, in runTask

return task.runDataRef(dataRef, **kwargs)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/19.0.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper

res = func(self, *args, **keyArgs)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/imageDifference.py”, line 719, in runDataRef

sensorRef.put(diaSources, self.config.coaddName + “Diff_diaSrc”)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0/python/lsst/daf/persistence/butlerSubset.py”, line 224, in put

self.butlerSubset.butler.put(obj, datasetType, self.dataId, doBackup=doBackup, **rest)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0/python/lsst/daf/persistence/butler.py”, line 1421, in put

locations = self._locate(datasetType, dataId, write=True)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0/python/lsst/daf/persistence/butler.py”, line 1297, in _locate

location = repoData.repo.map(datasetType, dataId, write=write)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0/python/lsst/daf/persistence/repository.py”, line 242, in map

loc = self._mapper.map(*args, **kwargs)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/19.0.0/python/lsst/daf/persistence/mapper.py”, line 166, in map

return func(self.validate(dataId), write)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_base/19.0.0/python/lsst/obs/base/cameraMapper.py”, line 387, in mapClosure

return mapping.map(mapper, dataId, write)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_base/19.0.0/python/lsst/obs/base/mapping.py”, line 155, in map

path = mapper._mapActualToPath(self.template, actualId)

File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_base/19.0.0/python/lsst/obs/base/mapping.py”, line 132, in template

“it must be set before it can be used.”)

RuntimeError: Template is not defined for the deepDiff_diaSrc dataset type, it must be set before it can be used.

imageDifference WARN: Could not persist metadata for dataId={‘filter’: ‘r2’, ‘tract’: ‘0’, ‘patch’: ‘0,0’, ‘extension’: 1, ‘visit’: 2426913, ‘runId’: ‘19AC24’, ‘object’: ‘SatEastDis’, ‘date’: ‘2019-07-01’, ‘ccd’: 0, ‘state’: ‘p’, ‘taiObs’: ‘2019-07-01T09:06:59.24’, ‘expTime’: 205.112}: Template is not defined for the deepDiff_metadata dataset type, it must be set before it can be used.

price · January 29, 2020, 12:24am

I expect this is a shortcoming in obs_cfht. We need to add deepDiff_diaSrc to obs_cfht/policy/MegacamMapper.yaml.

fraserw · January 29, 2020, 5:48pm

Thanks for the response @price .

For the sake of a project under time pressure, I will stick with using montage, and look again at this in a month or two. Thanks!

gkovacs · February 5, 2020, 2:35am

@fraserw I’m afraid subtraction of two skymap warped coadd/template is not supported in image differencing at all.

With getTemplate==GetCoaddAsTemplateTask (default), the following data handling happens in imageDifference.py:

ImageDifferenceTask gets the calexp with the --id specified from the Butler as the science exposure. I guess in your case, tract= and patch= numbers are ignored and only visit=, filter, date, ccd, extension, etc. may be used. The templateId option is ignored completely.
The butler picks the (first available) template (skymap) in the repository. The closest tract of the skymap is selected and the numbers of all overlapping patches within that tract.
Based on the science exposure (calexp) sky coordinates, and the tract WCS, image coordinates of a template exposure is determined, an empty exposure is created. (exposure dimensions=(2048, 4612); coadd dimensions=(1188, 2568))
Each existing coadd with the selected patch numbers are loaded and clipped into the template exposure. In other words, you have no command line control on which patches are loaded and used as template. (Reading patch...)
The template exposure is warped into the calexp then PSF matched and subtracted from the science calexp exposure.
The difference image is written out with the same dataId as the science calexp. I’m actually surprised that this is the first step to fail, I think because of the tract= patch= keys.

The templateId option is only used when 2 calexps are subtracted (getTemplate == GetCalexpAsTemplateTask), where it can contain keys (typically visit=) to update the dataId of the science exposure (–id) to get the dataId of the template calexp.

fraserw · April 29, 2020, 7:29pm

Hi @gkovacs

Thanks for the description.

Just getting back to this again. I haven’t been confidently able to parse what you described. From your fifth bullet, it sounds like imageDifference.py can remove a template image from a calexp image. Great. I should be able to project the subtracted images onto a rectalinear grid after the subtraction.

From what you describe, to control the image subtractoin, I need to make sure that only 1 template image exists in the butler. Right?