How to use ImageDifference.py

Support
,
#1

Hi, I’m trying to learn how to use the lsst.ip.diffim module for astronomical image differencing. In particular, I’m interested in the difference between calibrated exposures. To test the package, I used the sample of the HSC following the first two parts of the LSST pipelines tutorial, but I didn’t understand how to run the task.
With the Butler client from the lsst.daf.persistence module I searched what data IDs are available for the ‘calexp’ dataset type in a particular filter.

import lsst.daf.persistence as dafPersist
butler = dafPersist.Butler(inputs=‘DATA/rerun/processCcdOutputs’)
butler.queryMetadata(‘calexp’, [‘visit’, ‘ccd’], dataId={‘filter’: ‘HSC-R’})
[(903334, 16),
(903334, 22),
(903334, 23),
(903334, 100),
(903336, 17),
(903336, 24),
(903338, 18),
(903338, 25),
(903342, 4),
(903342, 10),
(903342, 100),
(903344, 0),
(903344, 5),
(903344, 11),
(903346, 1),
(903346, 6),
(903346, 12)]

Then, I shut down the Python interpeter and I typed:

imageDifference.py DATA --id visit=903334 ccd=22 --templateId visit=903336 ccd=24 --output imageDifference

And I got the error message:

CameraMapper INFO: Loading exposure registry from /home/vincenzo/Scrivania/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
root WARN: No data found for dataId=OrderedDict([(‘visit’, 903334), (‘ccd’, 22)])
root WARN: No data found for dataId=OrderedDict([(‘visit’, 903336), (‘ccd’, 24)])
root INFO: Running: /home/vincenzo/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/bin/imageDifference.py DATA --id visit=903334 ccd=22 --templateId visit=903336 ccd=24 --output imageDifference
WARNING: You are using OpenBLAS with multiple threads (8), but have not
specified the number of threads using one of the OpenBLAS environment variables:
OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
This may indicate that you are unintentionally using multiple threads, which may
cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
what you are doing and want threads enabled implicitly, set the environment
variable LSST_ALLOW_IMPLICIT_THREADS.
root WARN: Not running the task because there is no data to process; you may preview data using “–show data”

If I try to preview the data without doing the operation, I get:

imageDifference.py DATA --output imageDifference --show data
CameraMapper INFO: Loading exposure registry from /home/vincenzo/Scrivania/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3

imageDifference.py DATA --id visit=903334 ccd=22 --templateId visit=903336 ccd=24 --output imageDifference --show data
CameraMapper INFO: Loading exposure registry from /home/vincenzo/Scrivania/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
root WARN: No data found for dataId=OrderedDict([(‘visit’, 903334), (‘ccd’, 22)])
root WARN: No data found for dataId=OrderedDict([(‘visit’, 903336), (‘ccd’, 24)])

May someone help me? What I’m doing wrong?

#2

As a first guess, I think you’ll need to specify --rerun processCcdOutputs on your command line.

#3

Thank you for the answer. I’ve tried to specify the full path of the butler directory where the processed images are, but in that case it can’t find the calib directory.

imageDifference.py DATA/rerun/processCcdOutputs --output imageDifference --show data
CameraMapper INFO: Loading exposure registry from /home/vincenzo/Scrivania/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
HscMapper WARN: Unable to find calib root directory

Do you know if there is a more detailed guide on how to use the ip.diffim module?

#4

There’s a difference between DATA --rerun processCcdOutputs and DATA/rerun/processCcdOutputs: the former includes default calibrations under the DATA root, while the latter doesn’t.

(Well, I’m pretty sure of this but haven’t actually tested it, so sorry if that still doesn’t work.)

(The fact that calibration registries are found but data is not is a subtlety of the Gen2 Butler that should be going away soon.)

#5

Thank you,it worked! Now I understand better how to work with the Butler even if I have a follow up problem. I have created a skymap to perform image subtraction with:

makeDiscreteSkyMap.py DATA --id --rerun processCcdOutputs:imageDifference --config skyMap.projection=“TAN”

Then, I warped the exposures created by ProcessCcd.py onto the map with the makeCoaddTempExp.py task, following the tutorial part 4.

Now I can see what data I can process with

imageDifference.py DATA --rerun imageDifference --id --show data

However, when I try the subtraction between two images I obtain this error message

RuntimeError: doSelectSources=True and kernelSourcesFromRef=False,but template sources not available. Cannot match science sources with template sources. Run process* on data from which templates are built.

I think the whole procedure should be right. I haven’t controlled yet the HSC exposures of the tutorial and probably there aren’t different images of the same field which I can subtract. I will check with a correct set of source and template. Am I still missing something? Thank you again for the help!

#6

Hi, sorry if I return on this topic after a long time. I’m still trying to see the output of the image difference task, but the images of the tutorial doesn’t seem to be suitable for this purpose and if I try to use downloaded images of the same field from HSC I have a problem with the ingestion.

Do you know if there are any tested images to launch the task and see the output?

#7

Hi @Vincenzo, I just wanted to double check, did your question above about using HSC images with ImageDifference.py get re-raised and answered in your post here (HSC images ingestion for image differencing), or is the above still an issue? It looks like both posts are on the same issue but I wanted to make sure.

#8

Hi @MelissaGraham, I initially wrote my question here because it was related to the previous, but later I made a more general question in a new topic, so this post could actually be removed because it’s unnecessary.

1 Like
#9

Thanks for confirming, @Vincenzo . No need to remove, since we have the link to the thread that answers the question in order to redirect any folks who might find themselves on this thread. :+1:t2:

1 Like