How to use ImageDifference.py

imagedifference
hsc
Tags: #<Tag:0x00007f61a498c738> #<Tag:0x00007f61a498c5f8>

#1

Hi, I’m trying to learn how to use the lsst.ip.diffim module for astronomical image differencing. In particular, I’m interested in the difference between calibrated exposures. To test the package, I used the sample of the HSC following the first two parts of the LSST pipelines tutorial, but I didn’t understand how to run the task.
With the Butler client from the lsst.daf.persistence module I searched what data IDs are available for the ‘calexp’ dataset type in a particular filter.

import lsst.daf.persistence as dafPersist
butler = dafPersist.Butler(inputs=‘DATA/rerun/processCcdOutputs’)
butler.queryMetadata(‘calexp’, [‘visit’, ‘ccd’], dataId={‘filter’: ‘HSC-R’})
[(903334, 16),
(903334, 22),
(903334, 23),
(903334, 100),
(903336, 17),
(903336, 24),
(903338, 18),
(903338, 25),
(903342, 4),
(903342, 10),
(903342, 100),
(903344, 0),
(903344, 5),
(903344, 11),
(903346, 1),
(903346, 6),
(903346, 12)]

Then, I shut down the Python interpeter and I typed:

imageDifference.py DATA --id visit=903334 ccd=22 --templateId visit=903336 ccd=24 --output imageDifference

And I got the error message:

CameraMapper INFO: Loading exposure registry from /home/vincenzo/Scrivania/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
root WARN: No data found for dataId=OrderedDict([(‘visit’, 903334), (‘ccd’, 22)])
root WARN: No data found for dataId=OrderedDict([(‘visit’, 903336), (‘ccd’, 24)])
root INFO: Running: /home/vincenzo/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/bin/imageDifference.py DATA --id visit=903334 ccd=22 --templateId visit=903336 ccd=24 --output imageDifference
WARNING: You are using OpenBLAS with multiple threads (8), but have not
specified the number of threads using one of the OpenBLAS environment variables:
OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
This may indicate that you are unintentionally using multiple threads, which may
cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
what you are doing and want threads enabled implicitly, set the environment
variable LSST_ALLOW_IMPLICIT_THREADS.
root WARN: Not running the task because there is no data to process; you may preview data using “–show data”

If I try to preview the data without doing the operation, I get:

imageDifference.py DATA --output imageDifference --show data
CameraMapper INFO: Loading exposure registry from /home/vincenzo/Scrivania/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3

imageDifference.py DATA --id visit=903334 ccd=22 --templateId visit=903336 ccd=24 --output imageDifference --show data
CameraMapper INFO: Loading exposure registry from /home/vincenzo/Scrivania/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
root WARN: No data found for dataId=OrderedDict([(‘visit’, 903334), (‘ccd’, 22)])
root WARN: No data found for dataId=OrderedDict([(‘visit’, 903336), (‘ccd’, 24)])

May someone help me? What I’m doing wrong?


(K-T Lim) #2

As a first guess, I think you’ll need to specify --rerun processCcdOutputs on your command line.


#3

Thank you for the answer. I’ve tried to specify the full path of the butler directory where the processed images are, but in that case it can’t find the calib directory.

imageDifference.py DATA/rerun/processCcdOutputs --output imageDifference --show data
CameraMapper INFO: Loading exposure registry from /home/vincenzo/Scrivania/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/vincenzo/Scrivania/DATA/CALIB/calibRegistry.sqlite3
HscMapper WARN: Unable to find calib root directory

Do you know if there is a more detailed guide on how to use the ip.diffim module?


(K-T Lim) #4

There’s a difference between DATA --rerun processCcdOutputs and DATA/rerun/processCcdOutputs: the former includes default calibrations under the DATA root, while the latter doesn’t.

(Well, I’m pretty sure of this but haven’t actually tested it, so sorry if that still doesn’t work.)

(The fact that calibration registries are found but data is not is a subtlety of the Gen2 Butler that should be going away soon.)


#5

Thank you,it worked! Now I understand better how to work with the Butler even if I have a follow up problem. I have created a skymap to perform image subtraction with:

makeDiscreteSkyMap.py DATA --id --rerun processCcdOutputs:imageDifference --config skyMap.projection=“TAN”

Then, I warped the exposures created by ProcessCcd.py onto the map with the makeCoaddTempExp.py task, following the tutorial part 4.

Now I can see what data I can process with

imageDifference.py DATA --rerun imageDifference --id --show data

However, when I try the subtraction between two images I obtain this error message

RuntimeError: doSelectSources=True and kernelSourcesFromRef=False,but template sources not available. Cannot match science sources with template sources. Run process* on data from which templates are built.

I think the whole procedure should be right. I haven’t controlled yet the HSC exposures of the tutorial and probably there aren’t different images of the same field which I can subtract. I will check with a correct set of source and template. Am I still missing something? Thank you again for the help!