Task _metadata not found

I’m trying to create an empty Task to serve as a basis for further development. I registered the package but when I try to run it with this (with or without the --register-dataset-types option):
pipetask run --register-dataset-types -b $RC2_SUBSET_DIR/SMALL_HSC/butler.yaml -p $RC2_SUBSET_DIR/pipelines/DRP.yaml#mytask -i HSC/RC2/defaults -o u/something

I get the below error (Dataset type with name ‘mytask_metadata’ not found).

The three main classes look like this:

class MytaskConnections(pipeBase.PipelineTaskConnections,
                          dimensions=("tract", "patch", "skymap", "instrument", "visit"),
                          defaultTemplates={"coaddName": "deep",
                                            "skyWcsName": "jointcal",
                                            "photoCalibName": "fgcm",
                                            "calexpType": ""}):
    def __init__(self, **kwargs):

class MytaskConfig(pipeBase.PipelineTaskConfig, CoaddBaseTask.ConfigClass,
    def validate(self):

class MyTask(CoaddBaseTask):
    ConfigClass = MytaskConfig
    _DefaultName = "mytask"

    def __init__(self, **kwargs):

    def runDataRef(self, patchRef, selectDataList=[]):
        log.info("Starting runDataRef")
        self.run(None, None, None, 0, None)

    def runQuantum(self, butlerQC, inputRefs, outputRefs):
        log.info("Starting runQuantum")
        self.run(None, None, None, 0, None)

    def run(self, calExpList, ccdIdList, skyInfo, visitId=0, dataIdList=None, **kwargs):
        log.info("Starting run")

The error I get is this:

lsst.ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(MyTask, label=mytask) dataId={instrument: 'HSC', skymap: 'hsc_rings_v1', tract: 9813, patch: 46, visit: 358, ...}> failed; processing will continue for remaining tasks.
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+e6edad51e5/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 361, in _executeQuantaInProcess
    self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+e6edad51e5/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 144, in execute
    if self.checkExistingOutputs(quantum, butler, taskDef):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+e6edad51e5/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 363, in checkExistingOutputs
    existingRefs, missingRefs = findOutputs(self.skipExistingIn)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+e6edad51e5/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 354, in findOutputs
    ref = butler.registry.findDataset(
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/g0e0517256d+47328fee21/python/lsst/daf/butler/registries/sql.py", line 407, in findDataset
    storage = self._managers.datasets[datasetType.name]
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/g0e0517256d+47328fee21/python/lsst/daf/butler/registry/interfaces/_datasets.py", line 520, in __getitem__
    raise KeyError(f"Dataset type with name '{name}' not found.")
KeyError: "Dataset type with name 'mytask_metadata' not found."

This is done in the latest daily Docker image. How do I fix this?


This is confusing, because that --register-dataset-types option should have ensured that the dataset type was registered before it got to this point.

My best guess is that we’ve got a bug in which the butler used for execution (the one raising an exception here) is initialized before the dataset type registration happens in another butler, and that this is a stale cache problem (see the note here). If that’s the case, I think the problem will go away if you try again against the same data repository even without --register-dataset-types. Can you try that?

We’ve also been changing how the metadata dataset types in particular work a lot lately, to deal with some backwards compatibility changes, and it’s possible we’ve broken things more broadly somehow. But I wouldn’t expect the traceback you saw if that’s the case.

Oh, and as an side, you don’t need that runDataRef method at all - that’s purely a relic of our old middleware, and you’ll soon see it start to disappear from other tasks in the stack.

Thanks for the reply. It seems that this was caused by there not being any connections in the Connections class (thanks, Zach Langford).