I would like to run
ap_pipe.py using data that were already processed with
jointcal.py night by night since these are nights 1 year appart from each others, and so far proper motion is not handled.
I then use the
wcs produced by
jointcal to make night-by-night coadds.
So far I have managed to point to the template of night_1 with
ap_pipe using the
-- template option, to run DIA on the visits of night_2.
It works fine, but
ap_pipe re-processes the ‘ccdProcessing’ for night_2, which is time consuming and unnecessary since I already have these done.
I tried using
--reuse-outputs-from ccdProcessor, but I think this is only able to reuse the ouputs of an earlier run of
ap_pipe, and doesn’t look into the input path. I guess I could link the
calexps into the output path, but that is not ideal.
Is there a proper way of doing this?
And also, since I already have improved
wcs for these night_2 visits, that
jointcal generated. Is there a way to use them?
So technically I would like something like
ap_pipe.py Input_path --calib calib_path --template template_night1_path **--calexp calexp_night2_path --wcs wcs_night2_path** --output output_path -c diaPipe.apdb.db_url="sqlite:////link/to/association.db" -c diaPipe.apdb.isolation_level="READ_UNCOMMITTED" -c diaPipe.apdb.connection_timeout=240 --id visit=visitlist_night2
I managed to go around reprocessing
calexps by linking the outputs of
singleFrameDriver in the output of
ap_pipe before running.
It’s not ideal, but it works…
The problem with this is that I still don’t have an option to use the
As an alternative I was wondering if it is possible to use a ref catalog coming from
jointcal instead of panstarrs. That would have better astrometry than panstarrs when reprocessing the calexp, and the wcs created in that process would be kind of equivalent to the one returned by
As an additional follow up, I was wondering if there is a way to turn off the alert production. So far, it created files, like
alerts/*.avro that are really heavy. I don’t need them in my current analysis, so I would love to have an option to turn it off.
I see in the config file that there is
config.diaPipe.alertPackager.alertWriteLocation but that is only a way to decide where to output, I don’t see a way to turn it off…
I believe the only way for ap_pipe to use existing calexps is if they already exist in the output repository. If someone else knows better, please do chime in!
I don’t think there is a way to use jointcal WCS data products either, since ap_pipe is designed to run in “real time” and there will never be some external pre-existing better WCS.
Finally, as best I can tell, the “automatically write a boatload of humongous alerts” behavior is a quirk of release v20. Persisting alerts now defaults to False in the latest weekly build of diaPipe, but unfortunately that option didn’t exist yet in v20. You’ll either need to use a more recent ap_pipe or just delete them after the fact.
As Meredith says,
ap_pipe is not designed to take external WCS information, because that data will not be available to our alert pipeline in realtime processing of LSST camera data.
ap_pipe is a full pipeline for taking raw images as input and producing the full subtracted output.
ImageDifferenceTask is probably the part of this pipeline that you are looking for, and it takes the calexp to be differenced, so would not re-do the image processing.
A quick scan suggests that
ImageDifferenceTask itself does not have an option for an external WCS. This is something that we may want to use in future DRP processing, but it has is not a priority at the current time.
It is certainly possible to take an LSST-style output catalog and covert it to an LSST-style refcat, but I’m not aware of us having any current code to do that. You’d want to do it on a catalog produced from a coadded image to get the most depth.
lsst.afw.table.updateSourceCoords() will do an in-place update of the ra/dec coordinates of an in-memory
SourceCatalog using a SkyWcs (like that produced by jointcal), if you want to experiment with this on single images.