Image difference with different filters (HSC-R and R2)

Tags: #<Tag:0x00007fb37d931f38> #<Tag:0x00007fb37d931e20> #<Tag:0x00007fb37d931d08> #<Tag:0x00007fb37d931bf0>

Hi all,

We are having an issue to perform difference imaging with HSC R and R2 bands.
We are trying to look at transients in the 2016-2017 period, and we ideally want to form a template with good seeing pre-dating these data.
We thus generated a template using a coadd of the visits of pointing 01172 (2015-03-18) for filter R.

The issue is that the R filter was switched to R2 in-between.
When I try to perform image differencing (using diapipe’s imageDifferenceDriver.py), I get errors like

39112 WARN 2020-08-30T15:38:22.965+0200 imageDifferenceDriver.imageDifference.getTemplate ({'visit': 91552, 'pointing': 1793, 'filter': 'HSC-R2', 'ccd': 57, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2016-11-28', 'taiObs': '2016-11-28', 'expTime': 30.0})(getTemplate.py:270)- deepCoadd_sub, tract=9813, patch=40 does not exist

And indeed, if I try in a notebook:

import lsst.ip.diffim as I
mytask=I.GetCoaddAsTemplateTask()
calexprepo = '/sps/lsst/HSC/bracine/Pipeline_runs/SSP_UDEEP_COSMOS_20161101_20170501/DATA/rerun/coaddDriver_R_01172/'
b = Butler(calexprepo)
skymap = b.get('deepCoadd_skyMap')
exposure = b.get('calexp',dataId={'visit': 91552, 'pointing': 1793, 'filter': 'HSC-R2', 'ccd': 1})
tractInfo, patchList, skyCorners = mytask.getOverlapPatchList(exposure, skymap)
# returns an instance of lsst.daf.persistence.butlerSubset.ButlerSubset
availableCoaddRefs = dict()
for patchInfo in patchList:
    patchNumber = tractInfo.getSequentialPatchIndex(patchInfo)
    patchArgDict = dict(
        datasetType=mytask.getCoaddDatasetName() + "_sub",
        bbox=patchInfo.getOuterBBox(),
        tract=tractInfo.getId(),
        patch="%s,%s" % (patchInfo.getIndex()[0], patchInfo.getIndex()[1]),
        numSubfilters=mytask.config.numSubfilters,
    )
sensorRef = b.dataRef('calexp', dataId={'visit': 91552, 'pointing': 1793, 'filter': 'HSC-R2', 'ccd': 1})
sensorRef.datasetExists(**patchArgDict)

I get False, whereas for the same test on Z band, I get True.

I assume it is because the butler is looking for a coadd in the R2 band, which doesn’t exist.

Is there a way to go around this?
I saw that if I add
patchArgDict[‘filter’]=‘HSC-R’
then
sensorRef.datasetExists(**patchArgDict)
becomes True

Also is there a recommendation not to perform image differencing between R and R2 or I and I2?

Best,

Ben

I suspect there’s no way to get around this without making at least some modifications to the task itself, though there may be lower-level interfaces you could use that let you provide the template to use more directly. I don’t know much about ImageDifferenceTask personally, so I hope someone who does can chime in on that possibility.

As for your last question:

Also is there a recommendation not to perform image differencing between R and R2 or I and I2?

There are significant differences in transmission as a function of wavelength between the versions of the two filters, so there’s at least some danger of getting some “spurious” detections that are really just static objects with SEDs that are sensitive to those differences.

But The R2 and I2 filters exist because the transmission curves of the R and I filters depend strongly on radius, and those internal differences are about as big as the average different between R and R2 or I and I2. So if you really want to avoid problems, you’d need to stick to just R2 and I2, or only work with R and I data with very small dithers (as is true the HSC SSP UltraDeep fields).

I don’t know of any quantitative analysis of the impact of the HSC transmission curve differences on transient detection, though; this is all just making guesses based on what the transmission curves look like.

This is probably a highly unsatisfactory answer, but the gist is that ImageDifferenceTask currently doesn’t support using a template and a science image which have different filters. There is probably a way to hack getTemplateTask in ip_diffim to accept a mismatch filter, but I got around this myself by just building an R2 and an I2 template instead of trying to force it to accept R and I. Half because it was easier and half because of the actual physical filter differences Jim mentions above.

If you do decide to hack around and come up with something that accepts different filers, please do let us know - I can certainly see this being a reasonable use case for various situations, but it’s not at the top of anyone’s priority list for now.