Task _metadata not found

Data Management
#1

Hi,
I’m trying to create an empty Task to serve as a basis for further development. I registered the package but when I try to run it with this (with or without the --register-dataset-types option):
pipetask run --register-dataset-types -b $RC2_SUBSET_DIR/SMALL_HSC/butler.yaml -p $RC2_SUBSET_DIR/pipelines/DRP.yaml#mytask -i HSC/RC2/defaults -o u/something

I get the below error (Dataset type with name ‘mytask_metadata’ not found).

The three main classes look like this:

class MytaskConnections(pipeBase.PipelineTaskConnections,
                          dimensions=("tract", "patch", "skymap", "instrument", "visit"),
                          defaultTemplates={"coaddName": "deep",
                                            "skyWcsName": "jointcal",
                                            "photoCalibName": "fgcm",
                                            "calexpType": ""}):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

class MytaskConfig(pipeBase.PipelineTaskConfig, CoaddBaseTask.ConfigClass,
                     pipelineConnections=MytaskConnections):
    def validate(self):
        super().validate()

class MyTask(CoaddBaseTask):
    ConfigClass = MytaskConfig
    _DefaultName = "mytask"

    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def runDataRef(self, patchRef, selectDataList=[]):
        print("runDataRef")
        log.info("Starting runDataRef")
        self.run(None, None, None, 0, None)

    def runQuantum(self, butlerQC, inputRefs, outputRefs):
        print("runQuantum")
        log.info("Starting runQuantum")
        self.run(None, None, None, 0, None)

    def run(self, calExpList, ccdIdList, skyInfo, visitId=0, dataIdList=None, **kwargs):
        log.info("Starting run")

The error I get is this:

lsst.ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(MyTask, label=mytask) dataId={instrument: 'HSC', skymap: 'hsc_rings_v1', tract: 9813, patch: 46, visit: 358, ...}> failed; processing will continue for remaining tasks.
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+e6edad51e5/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 361, in _executeQuantaInProcess
    self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+e6edad51e5/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 144, in execute
    if self.checkExistingOutputs(quantum, butler, taskDef):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+e6edad51e5/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 363, in checkExistingOutputs
    existingRefs, missingRefs = findOutputs(self.skipExistingIn)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/ctrl_mpexec/g2085e88109+e6edad51e5/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 354, in findOutputs
    ref = butler.registry.findDataset(
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/g0e0517256d+47328fee21/python/lsst/daf/butler/registries/sql.py", line 407, in findDataset
    storage = self._managers.datasets[datasetType.name]
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/g0e0517256d+47328fee21/python/lsst/daf/butler/registry/interfaces/_datasets.py", line 520, in __getitem__
    raise KeyError(f"Dataset type with name '{name}' not found.")
KeyError: "Dataset type with name 'mytask_metadata' not found."

This is done in the latest daily Docker image. How do I fix this?

Thanks!
Petar

#2

This is confusing, because that --register-dataset-types option should have ensured that the dataset type was registered before it got to this point.

My best guess is that we’ve got a bug in which the butler used for execution (the one raising an exception here) is initialized before the dataset type registration happens in another butler, and that this is a stale cache problem (see the note here). If that’s the case, I think the problem will go away if you try again against the same data repository even without --register-dataset-types. Can you try that?

We’ve also been changing how the metadata dataset types in particular work a lot lately, to deal with some backwards compatibility changes, and it’s possible we’ve broken things more broadly somehow. But I wouldn’t expect the traceback you saw if that’s the case.

#3

Oh, and as an side, you don’t need that runDataRef method at all - that’s purely a relic of our old middleware, and you’ll soon see it start to disappear from other tasks in the stack.

#4

Thanks for the reply. It seems that this was caused by there not being any connections in the Connections class (thanks, Zach Langford).