I am working on running DECam data through the DRP-Merian pipeline and am running into an issue on the first step of the DRP. I have a handful of exposures whose visits are only partially-covered by the photom (ps1) and astrom (gaia) reference catalogs that I currently have downloaded. From what I can tell, when I am trying to run DRP#step1, the qgraph prevents pipetask (or bps) from proceeding any further when there are no sources in my reference catalogs covering a given CCD. Specifically the error from when the quantum graph is formed is:
FileNotFoundError: Not enough datasets (0) found for non-optional connection calibrate.astromRefCat (gaia) with minimum=1 for quantum data ID {instrument: 'DECam', detector: 45, visit: 662919, ...}
Is there any way I can force the qgraph to skip the these specific detectors without sources in the catalog, other than passing specific data queries via -d?
Thanks for following up on this, I have worked through the single frame processing before but unfortunately the tutorial outlines using the -d command to query different datasets, which I am trying to avoid since the WCS of individually CCDs hasn’t been fully calibrated at this stage of processing (in principle it is still possible to run through with a script ahead of time to remove visits/exposures outside of the reference catalogs by estimating the angular distance between a given CCD and the current target Ra/Dec… but this will also cost exposure time at the edges of my overall fov).
Did you mean using the single-frame processing pipeline used in the tutorial for HSC data? That has not been implemented in the DECam DRP releases… but looking at the .yaml’s the individual tasks are identical to DRP-Merian#step1.
One additional update, with source catalogs covering a larger FOV the issue is resolved… but it would still be nice to have a solution to this problem for objects which lack good coverage from my photometric reference catalogs.
To ignore those detectors and proceed, what you’ll want to do is let that run with the failures run to completion. Unless you’re running with some kind of “fail fast” option (I’m not sure we even have one), that will naturally finish any tasks that are not blocked by the tasks that failed.
And then you can just proceed to running step2a against those outputs; that’s where the first visit-wide gather tasks appear, and QG generation will naturally skip the detectors for which the inputs (i.e. the outputs of step1) are not available.
Interesting, currently I do not have the “fail fast” option enabled, but the job is still exiting at the qgraph stage, maybe it has to do with how I am submitting it? Currently I am using bps submit, e.g.
@jbosch certainly knows better than I, but I think the issue here is that the refcats are PrerequisiteInput, so if one isn’t found, the Quantum Graph generation fails? The only way I currently see to possibly avoid this is by setting the doAstrometry/Photometry configs to False:
But then you don’t get those calibrations run for anything, so this is certainly not ideal! Perhaps we need an option that allows the graph generation to just skip any detectors for which is can’t find an associated refcat?
That sounds reasonable (from a non-LSP expert)… in that case maybe the fix is as simple as adding an extra config option which passes one (or both) of these checks. In principle that’s simple enough to hard-code on my local LSP just to check… I’ll give this a try tomorrow and update as necessary
I think a feature like this will be really valuable for groups like mine which, generally speaking, are interested in only studying portions of the sky and use/ingest partial reference catalogs as a result. In my case, for cluster WL, we’re generally only interested in a ~1.5deg x 1.5deg square centered at the cluster, so if detectors near the edges of a visit are dropped due to a lack of refcat coverage, we’d still like to use the detectors which are covered.
@laurenam is right; I was guessing that the problem was that you just didn’t get enough reference stars at runtime, rather than some detectors not overlapping a reference catalog shard at all.
It might work to add minimum=0here and here in a local copy of pipe_tasks. This would let QG generation succeed and probably lead to a runtime failure for those detectors. We wouldn’t want to make that change on main, but there’s a more complicated version of this change that would set the minimum to zero if a config option is set that we could consider.
I’ll weigh in that although the suggestion proposed by @jbosch above is a reasonable fix for a user, I strongly think that we should not include such functionality in our main pipeline. CalibrateImageTask intentionally is not as configurable in this manner, so that it is easier to reason about its functionality, and we are working on replacing CharacterizeImageTask/CalibrateTask with this new Task (AP should be doing so early next week).