I have recently upgraded to v29.0 and am testing it to process data with a customized obs_ package.
I have run through these tasks successfully:
- isr
- calibrateImage
- transformPreSourceTable
- consolidatePreSourceTable
- consolidateVisitSummary
- isolatedStarAssociation
- jointcal
and got an error when I tried to generate the SourceTable with the following tasks:
- finalizeCharacterizationDetector
- consolidateFinalizeCharacterization
- updateVisitSummary
- writeRecalibratedSourceTable
- transformSourceTable
- consolidateSourceTable
(I also tried finalizeCharacterization instead of finalizeCharacterizationDetector, but got the same error.)
Here is the error message:
lsst.pipe.base.quantum_graph_builder INFO: Processing pipeline subgraph 1 of 2 with 2 task(s).
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File "/soft/byliu/lsstsw/stack/lsst-scipipe-10.0.0/Linux64/ctrl_mpexec/ge10c2aeecd+a63486ee86/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 230, in run
if (qgraph := script.qgraph(pipeline_graph_factory, **kwargs, show=show)) is None:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/soft/byliu/lsstsw/stack/lsst-scipipe-10.0.0/Linux64/ctrl_mpexec/ge10c2aeecd+a63486ee86/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 231, in qgraph
qgraph = f.makeGraph(pipeline_graph_factory, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/soft/byliu/lsstsw/stack/lsst-scipipe-10.0.0/Linux64/ctrl_mpexec/ge10c2aeecd+a63486ee86/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 707, in makeGraph
qgraph = graph_builder.build(metadata, attach_datastore_records=args.qgraph_datastore_records)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/soft/byliu/lsstsw/stack/lsst-scipipe-10.0.0/Linux64/utils/gcdad8030b8+8c5ae1fdc5/python/lsst/utils/timer.py", line 308, in timeMethod_wrapper
res = func(self, *args, **keyArgs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/soft/byliu/lsstsw/stack/lsst-scipipe-10.0.0/Linux64/pipe_base/ge97eb97271+45990178f5/python/lsst/pipe/base/quantum_graph_builder.py", line 344, in build
subgraph_skeleton = self.process_subgraph(subgraph)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/soft/byliu/lsstsw/stack/lsst-scipipe-10.0.0/Linux64/utils/gcdad8030b8+8c5ae1fdc5/python/lsst/utils/timer.py", line 308, in timeMethod_wrapper
res = func(self, *args, **keyArgs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/soft/byliu/lsstsw/stack/lsst-scipipe-10.0.0/Linux64/pipe_base/ge97eb97271+45990178f5/python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py", line 141, in process_subgraph
self._query_for_data_ids(tree)
File "/soft/byliu/lsstsw/stack/lsst-scipipe-10.0.0/Linux64/pipe_base/ge97eb97271+45990178f5/python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py", line 175, in _query_for_data_ids
raise QuantumGraphBuilderError(
lsst.pipe.base.quantum_graph_builder.QuantumGraphBuilderError: No datasets for overall-input 'recalibrated_star_unstandardized' found (the dataset type is not even registered). This is probably a bug in either the pipeline definition or the dataset constraints passed to the quantum graph builder.
I don’t think I have seen this “recalibrated_star_unstandardized” dataset in the single-frame or jointcal tasks. Since this workflow works well on an older version (eg. v26.0), I wonder if I missed some new necessary tasks after jointcal.
(I am using jointcal rather than fgcm/gbdes because I have not figured out the required configs for fgcm, and at this time I’m just trying to make some single-band coadds.)
Please let me know if there is any other info I can provide. Thanks!