Hi everyone!
I am a complete newbie using th pipeline and I’m trying to do image difference on some decam data, but I’m not quite sure how to use it. This is what I tried:
$ imageDifference.py DATA_Javiera/pointing1/ --output output1
CameraMapper INFO: Loading exposure registry from /mnt/flock/jahumada/DATA_Javiera/pointing1/registry.sqlite3
root INFO: Running: /home/jahumada/lsst_stack-v21/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/bin/imageDifference.py DATA_Javiera/pointing1/ --output output1 conda.common.io INFO: overtaking stderr and stdout conda.common.io INFO: stderr and stdout yielding back
root WARN: Not running the task because there is no data to process; you may preview data using “–show data”
imageDifference.py DATA_Javiera/pointing1/ --output output1 --show data
CameraMapper INFO: Loading exposure registry from /mnt/flock/jahumada/DATA_Javiera/pointing1/registry.sqlite3
But I don’t know why it doesn’t seem to be able to access the data, since it is in that folder (DATA_Javiera/pointing1). Maybe I skipped some previous steps?
The only thing I did before was to ingest the instcal data.
If someone has an idea why this is happening and points it out to me, I would really appreciate it
The error message is because there is no --id argument in your command line. One very unintuitive aspect of these scripts is the need to always provide --id, even if you intend to process all data (in which case, --id need not be followed by anything).
I am unfortunately not familiar with instcals, but I don’t think they can be differenced immediately after ingesting. I think you need to first run processCcd.py by creating a config override file as described in steps 6 and 5 (in that order!) of the obs_decam readme. You can then run imageDifference.py on the output of processCcd.py.
Hi Javiera, it sounds like you ingested some DECam instcals into an old-style “gen2” repository and would like to use it to run image differencing. Unfortunately there are several intermediate steps, and we are moving away both from supporting instcals and toward a new “gen3” system for running tasks like imageDifference.
In general, you need to run “processCcd” (isrTask, characterizeImageTask, and calibrateTask), as well as a set of tasks to make coadd images for use as templates (e.g., makeWarpTask and assembleCoaddTask), before you have the input data products needed to run imageDifferenceTask.
If you’re working with instcals, you will want to use DecamNullIsrTask (which lives in the obs_decam package) instead of the regular isrTask. You can choose to turn off astrometric and photometric calibration in calibrateTask if you don’t want to supply external reference catalogs, but this is not recommended if you want scientifically valid results. After “processCcd” you will have a data product Krzysztof referred to, a “calexp.” This is the input imageDifferenceTask expects. I believe it is still possible to configure the task to treat one calexp as “science” and another as “template,” but the default assumes you have a stacked “coadd” data product to use as a template, which is how we will do difference imaging with actual Rubin data.
I realize this is not a straightforward answer. I’d encourage you to check out the Stack Club resources if you haven’t, to consider starting over with our newer gen3 middleware system (which does not support instcals), and to start from raws (instead of instcals) and build your own “master” calibration frames with the cp_pipe package for use during ISR.
Hi Meredith, thanks a lot for your output!. I’m in a similar situation as Javiera, using a gen2 repository. I was wondering how recommendable is to keep going forward with this? or is it better to just start over again with the raw images. Another question related is, if you know for how much time will gen2 repository be supported? it doesn’t have to be an exact answer an estimation can help. Thanks a lot again!
Gen2 will be removed sometime after we release v23.0, and no earlier than the end of 2021. It will no longer be maintained beginning January 2022. I know switching over is a lot of work, but I do think it is best to start over with raws and try Gen3 if you can, especially if you plan to use the dataset for more than 6 months or so. One bonus is that folks on this forum are more likely to eagerly help with Gen3-related questions
Hi! I was wondering what was the reason behind not supporting instcal type images and moving on to raw images. I heard through other scientists that the instcal images from DECAM are reliable to their science, but maybe when it comes to Image Differencing, that is no longer true.
If there are any references that could help me understand this change, I would gladly receive it
A fair question. In general, the purpose of the Science Pipelines is to process Rubin data (once it exists), and all other datasets are to prepare for that. Since we will have to build our own calibrated images from scratch each time, we don’t generally prioritize supporting other formats. That’s not to say we won’t ever support them or they are bad, just that we must build a system that does all the processing correctly without assuming we have access to an instcal-like product.
an estimation can help. Thanks a lot again!