I took a quick look at this. Default A&L differencing sets doDecorrelation to True, so that explains why your first case matches your third case. In addition, it appears that as of about a year ago, doPreConvolvewas deprecated and now one must set useScoreImageDetection instead - this explains your second case.
If it’s any consolation, we are very close to completing a long-overdue full refactor of imageDifferenceTask! You can see the details on DM-33001. That ticket is “Done,” but we have not yet adopted the new task(s) in our default AP or DRP pipelines - testing and validation is still in progress.
Hi! I encountered an error that I’m not sure how to work around.
setting useScoreImageDetection as True will rise the following error:
ValueError: doWriteSubtractedExp=True and useScoreImageDetection=True Regular difference image is not calculated. AL subtraction calculates either the regular difference image or the score image.
but setting useScoreImageDetection as True and doWriteSubtractedExp as False results in an empty QuantumGraph:
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'Coadd_AGN/Blind15A_16_2exp'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'processCcdOutputs/Blind15A_16'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'refcatsv2'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'DECam/raw/all'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'DECam/calib'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'skymaps'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'DECam/calib/20180504calibs'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'DECam/calib/20150219calibs'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'DECam/calib/20150220calibs'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'DECam/calib/20150218calibs'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'DECam/calib/20150217calibs'.
lsst.pipe.base.graphBuilder WARNING: No datasets of type goodSeeingDiff_differenceExp in collection 'DECam/calib/20150221calibs'.
py.warnings WARNING: /home/jahumada/lsstw_v22_10/lsst-w.2022.10/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/ctrl_mpexec/g34d403d65c+f0cf790be1/python/lsst/ctrl/mpexec/cli/script/qgraph.py:183: UserWarning: QuantumGraph is empty
qgraph = f.makeGraph(pipelineObj, args)
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File "/home/jahumada/lsstw_v22_10/lsst-w.2022.10/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/ctrl_mpexec/g34d403d65c+f0cf790be1/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 126, in run
qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
File "/home/jahumada/lsstw_v22_10/lsst-w.2022.10/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/ctrl_mpexec/g34d403d65c+f0cf790be1/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 186, in qgraph
raise RuntimeError("QuantumGraph is empty.")
RuntimeError: QuantumGraph is empty.
Not sure how I should proceed , I bet there’s a parameter that I should consider, but I don’t know which.
I find it a bit weird that both of these parameters cannot be True at the same time. I understand that doWriteSubtractedExp writes/registers the difference images…
I haven’t played much with these specific configs and don’t have any insights, sorry. I think the newly refactored ImageDifference I mentioned before is probably going to be easier to switch to than sorting out the idiosyncrasies of every possible config combination in “old” ImageDifference. Is there something in particular that isn’t working how you’re hoping for with the default configs?
I’m currently comparing the light curves I get from Image Differencing versus the light curves obtained by aperture photometry for the same sources. I expect both of these to have a similar structure, that is, the same tendencies.
Unfortunately, there are many cases where both of these LCs have a significantly different nature, and it so happens to be for the sources where the subtraction is worse. With the latter, I mean that there are big flux dipoles.
I was hoping that doing pre-convolution to the science images may help with these dipoles.
I assume that I can access this refactored ImageDifference with the newest release of the pipeline?
In principle yes, though we are still testing it you can follow along in semi-real-time here. Very heavy disclaimers, there are almost certainly some bugs we haven’t squashed yet - in particular, the science image pre-convolution option is giving us some weird patch edge artifacts at this moment. I defer to @kherner for specific pointers he may wish to give. It may be best to give us a bit to iron out the rough parts of science image pre-convolution before diving in.
As @mrawls mentioned we’re still doing some evaluation and commissioning studies, but if you don’t want to wait for those, the new code will certainly run. I’m not sure which pipeline you’re exactly using, but the idea is to replace the imageDifference with two new tasks: subtractImages and detectAndMeasure. If you want to do a test with the defaults, a minimally-sufficient modification to your pipeline would be something like this:
The parts of your pipeline before and after imagedifference would remain the same (note you may need to modify some contract names in that case). If you already have an imageDifference task in your pipeline, you will need to exclude that. In this example I am using parameters.coaddName as defined in the ApPipe.yaml in the ap_pipe package; your pipeline might be slightly different. If you want to run the new tasks in compatibility mode, which should give you results very close to the existing imageDifference task, you can add
forceCompatibility: True
to the config section of the subtractImages task. Let us know if you run into any difficulties.
Thanks Ken, I will now be trying the configs you suggest!
I have a question now specific to the code of Image Subtraction. From reading the available documentation on the implementation code of Alard and Lupton in the pipelines, I’m not clear about the treatment of scaling the flux of the template (coadd) image to match the science image.
Checking the template (R), science (I) and difference image, I see that the three of them have different calibration zero points. I understand that the template is convolved with a kernel k that minimizes the equation R Ⓧ k - I. Does the kernel account for the mismatch in flux between the template and the science image? And if it does, is the convoluted template saved by the pipeline at any point?
I assume that the photometric zero-point mismatch between the science and difference Image is due to the latter showing fainter signals.
I wonder where I should add the configuration of pre-convolution of the science images with their own psf. I checked that in subtractImages.py there’s a function called finalize:
If you look at the docstring for the finalize function you’ll see that the preConvMode option is just telling the function whether the science image was pre-convolved or not; it doesn’t actually do the pre-convolution itself. The new diffim workflow doesn’t actually do it yet, though it is still in the plans.
I see… I would really like to try to pre-convolute my science images, though! to check if, by doing so, the flux dipoles for some of the sources minimize…
If there’s any mock/preliminary script that could do the pre-convolution, I would happily try it out. Maybe using the _convolveExposure function in a prior step is feasible?
At the moment there is unfortunately no version of image differencing that will properly support preconvolution. If you just want to see what the images would look like with preconvolution, you could an extra convolution step as you suggest, though you would also need to measure the PSF of the convolved image (or analytically calculate the new PSF FWHM) in order to get the right basis functions for the matching kernel. Also note that this would result in an incorrect mask and variance plane, so if you try to run any of the rest of the pipeline after image subtraction you will not get reasonable results. We plan to support preconvolution with the new image differencing in the next few months.
Thanks Ian
a question about the deepDiff_matchedExp, this image should correspond to the astrometrically aligned, convolved, and scaled template that matches the psf and flux counts of the science image to which it’s later subtracted, right?
Yes, that’s correct. Except that the flux calibration is not automatically applied. You will still have to use the photoCalib attached to the exposure to apply the photometric calibration.
Thanks Ian a question about this. So, photocalib would transform the ADU counts to nJy flux by multiplying the former to a scaling number obtained from the stars selected by the pipeline. But does photocalib account for the zero-point of the images as well? or should that be applied separately?
The PhotoCalib object converts from instrumental counts to nanoJansky. The zero-point of the images is irrelevant, since the resulting flux in nJy is the actual flux on-sky, whether in the science, difference, or template image.
If you’re wondering about surface brightness images vs. fluence images, we don’t currently deal with that distinction in a coherent manner; see the discussion here for more: Photometric calibration and astrometric Jacobians
, I bet there’s a parameter that I should consider, but I don’t know which.
you can follow along in semi-real-time 