Specifically, I’m trying to run the following command to process a small amount of the data:
pipetask run -b $RC2_SUBSET_DIR/SMALL_HSC/butler.yaml
-p $RC2_SUBSET_DIR/pipelines/DRP.yaml#singleFrame
-i HSC/RC2/defaults
-o u/$USER/single_frame
–register-dataset-types
-d “instrument=‘HSC’ AND detector=41 AND exposure=322”
I do get some logging output that suggests the pipeline is running and completing at least some initial steps. But ultimately the pipeline seems to fail with an error:
RuntimeError: No good pixels in image array
I’m using pipeline version w_2022_06.
$ eups list lsst_distrib
g2d4714e03a+1ca36dd228 current w_2022_06 w_latest setup
I’ve attached the full log corresponding to the pipetask command I ran. Thanks very much for any advice._example_41.log (22.5 KB)
Hi Aaron. That’s strange that it’s failing when trying to calculate a sky background correction. I’m not certain whether the tutorial was intended to include the sky correction step, but you could try using the “simpleSingleFrame” subset instead of “singleFrame” as a way forward (i.e., pipelines/DRP.yaml#simpleSingleFrame instead of pipelines/DRP.yaml#singleFrame). This skips a couple of steps from the overall data processing pipeline, but I don’t think it will affect any of the later tutorial steps.
I think the original command should have worked, but I’ve tested it and got the same result you did. So I think you’ve come across a real bug. From what I can tell, the pipeline should be fine without the tasks that are excluded by simpleSingleFrame, but definitely holler if that’s not true!
lsst.pipe.base.graphBuilder WARNING: Dataset type visitSummary is not registered.
lsst.pipe.base.graphBuilder WARNING: Dataset type sourceTable_visit is not registered.
And ultimately an error:
RuntimeError: QuantumGraph is empty.
I’ve attached the full log.
It doesn’t seem especially likely to me that visit summary information would be part of what’s dropped in the replacement of singleFrame with simpleSingleFrame, but I’m not an expert. Searching around online, I found docs suggesting that ConsolidateSourceTableTask creates sourceTable_visit and ConsolidateVisitSummaryTask creates visitSummary. Do I need to run those tasks in between part 2 and part 4 of the tutorial? If so, how? Thanks again, I really appreciate the help.
Apologies for the delay in responding, Aaron. You’ve uncovered some misleading advice we’re giving in the tutorial, so it took a bit to sort out what’s amiss.
The simple answer to your original issue is that the tutorial as written will not successfully execute if you follow the advice in the callout to limit to only a single detector/exposure. That will indeed show you quickly what it looks like when you execute a pipetask, but because it only spans a single detector, it does not contain enough sky area for the skyCorr function to fit the sky background. One solution is to simply leave out the selection of the single detector and exposure (i.e., leave out the data query specified by the “-d…” syntax), and execute the commands on all data in the repository. That will take some time (possibly a few hours), but I have confirmed that the steps as outlined work fine if you use the full dataset.
It is likely that you could successfully use a subset of exposures, but I haven’t yet been able to confirm a minimal working subset. I’ll let you know (and get the tutorial website updated) if I find a smaller subset to use.
Thanks, Jeff! In my post from Feb 17, I seem to have claimed that I ran all of the simpleSingleFrame processing, but perhaps I mistakenly did something like limiting to a specific exposure and/or detector. I’ve started completely fresh and am now running:
Hi Aaron - you’ll want to use the singleFrame subset rather than simpleSingleFrame. The additional steps that are included in singleFrame produce source and summary tables that are needed by FGCM (as you saw in your previous attempt). Apologies for leading you astray by suggesting simpleSingleFrame initially.
Looks like I switched to using the w_2022_09 version of the stack, but w_06 should work fine.
Oh, perhaps I was unclear. The singleFrame subset will not run on a single detector/exposure, because the skyCorr task requires a larger contiguous area. If you want to complete the entire tutorial, you will need to run singleFrame with the full rc2_subset dataset.
Thanks for reiterating that, Jeff. I’ve now been able to reach the warping stage of the tutorial. I can add to this thread if I have any further questions. Thanks again!