imageDifference products in HscMapper.yaml

Support
#1

Hi!

I am trying to run imageDifference.py on HSC images with HSCpipe 8.4. I am subtracting individual exposures from coadded templates. I am confused by entries like the following in HSCMapper.yaml regarding difference imaging products:

deepDiff_differenceExp:
    template: 'deepDiff/%(pointing)05d/%(filter)s/DIFFEXP-%(visit)07d-%(ccd)03d.fits'

I think this mapper needs to have a tract entry like following:

deepDiff_differenceExp:
    template: 'deepDiff/%(pointing)05d/%(filter)s/tract%(tract)d/DIFFEXP-%(visit)07d-%(ccd)03d.fits'

This is because imageDifference.py requires specification of the tract and the differencing is done tract by tract. In case there is a ccd chip which falls in an overlap region between tracts, depending upon which tract was differenced first, the second one can overwrite the files produced by the first one.

Has anyone experienced this before?

I think the simple fix like what is shown above to alll of the difference imaging products in the HscMapper.yaml file will solve the problem.

#2

Hi Surhud,

My experience with the imageDifference process is that each single image has a mosaic of ‘tract’ based coAdds subtracted from it. This happens in a single process. The coAdd tracts are each subtracted from the CCD based single image and written to disk when the fully subtracted image has been produced. Thus, there is no ‘tract’ in the deepDiff name.

I am curious to know if there are other processes for imageDifference that do result in tract based single differenced images.

JJ

#3

Note that the Mapper yaml files like that one are part of the gen2 butler, which is now deprecated.
https://pipelines.lsst.io/releases/v22_0_0.html#release-v22-0-0-deprecate-gen2

I believe that the problem related to detectors falling in a tract overlap region is being fixed on DM-29310, but only in gen3 (gen2 support is phasing out).

#4

@jjkavelaars I am not sure that this is correct. The imageDifference.py proceeds in a tract by tract fashion, and assembles the template that overlaps with the science image per tract. If there are multiple patches in a tract that overlaps with a given science image, it will deal with it correctly, but the imageDifference.py script does not assemble the template from patches which come from different tracts. This is the root cause of the overwriting issue. Once I fixed the Mapper yaml file I was able to see two different difference images one showing half of the image correctly subtracted, another one showing the other half which overlaps with the corresponding tract.

#5

Thanks @parejkoj . I guess @rearmstr also ran in to a similar issue. I suppose the fix I mentioned in the original post should be reasonable, right?

#6

For DESC we wrote some code to combine all overlapping tracts into a single template image. In this way we did not have to specify a tract when running imageDifference.py. The code is here dia_pipe/getMultiTractTemplate.py at master · LSSTDESC/dia_pipe · GitHub.

You can then retarget the code like task like this:
config.imageDifference.getTemplate.retarget(GetCoaddAsMultiTractTemplateTask

I think it should work in your case.

#7

Thanks @rearmstr! I will try to use this with hscPipe 8.4.