afwTables: A Guided Tour - No value in data ID error

I’ve been using some of Imran Hasan’s Stack Club tutorial but last week it failed, and now I see his code fails as well. Could it be an issue with w_2021_49? :
[60] # For the rest of this tutorial, we are going to work with coadd data products
tract = 4851
patch = 29 #gen3 analog to gen2 patch id ‘1,4’
bandList = [“i”, “r”]
filterList = bandList # using filterList for consistency with cells folllowing below
[61] objects = []
for filter in filterList:
dataId = {‘band’:filter, ‘tract’:tract, ‘patch’:patch}
objects.append(butler.get(“deepCoadd_forced_src”, dataId=dataId))
iSources, rSources = objects


KeyError Traceback (most recent call last)
/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-110-g1427568b+500492d978/python/lsst/daf/butler/core/dimensions/_coordinate.py in standardize(mapping, graph, universe, defaults, **kwargs)
233 try:
→ 234 values = tuple(d[name] for name in graph.required.names)
235 except KeyError as err:

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-110-g1427568b+500492d978/python/lsst/daf/butler/core/dimensions/_coordinate.py in (.0)
233 try:
→ 234 values = tuple(d[name] for name in graph.required.names)
235 except KeyError as err:

KeyError: ‘skymap’

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
/tmp/ipykernel_3588/2849544431.py in
2 for filter in filterList:
3 dataId = {‘band’:filter, ‘tract’:tract, ‘patch’:patch}
----> 4 objects.append(butler.get(“deepCoadd_forced_src”, dataId=dataId))
5 iSources, rSources = objects

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-110-g1427568b+500492d978/python/lsst/daf/butler/_butler.py in get(self, datasetRefOrType, dataId, parameters, collections, **kwargs)
1160 “”"
1161 log.debug(“Butler get: %s, dataId=%s, parameters=%s”, datasetRefOrType, dataId, parameters)
→ 1162 ref = self._findDatasetRef(datasetRefOrType, dataId, collections=collections, **kwargs)
1163 return self.getDirect(ref, parameters=parameters)
1164

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-110-g1427568b+500492d978/python/lsst/daf/butler/_butler.py in _findDatasetRef(self, datasetRefOrType, dataId, collections, allowUnresolved, **kwargs)
902 # type instead of letting registry.findDataset do it, so we get the
903 # result even if no dataset is found.
→ 904 dataId = DataCoordinate.standardize(dataId, graph=datasetType.dimensions,
905 defaults=self.registry.defaults.dataId, **kwargs)
906 # Always lookup the DatasetRef, even if one is given, to ensure it is

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-110-g1427568b+500492d978/python/lsst/daf/butler/core/dimensions/_coordinate.py in standardize(mapping, graph, universe, defaults, **kwargs)
234 values = tuple(d[name] for name in graph.required.names)
235 except KeyError as err:
→ 236 raise KeyError(f"No value in data ID ({mapping}) for required dimension {err}.") from err
237 # Some backends cannot handle numpy.int64 type which is a subclass of
238 # numbers.Integral; convert that to int.

KeyError: “No value in data ID ({‘band’: ‘i’, ‘tract’: 4851, ‘patch’: 29}) for required dimension ‘skymap’.”

After the butler is initialized in the notebook, but before the cell where this error occurs, could you add:

print(butler.registry.defaults.dataId)
print(butler.registry.defaults.collections)

If you have a way to do the same with a version of the stack where this worked, it’d also be useful to compare the two.

Hi Jim!
{instrument: ‘LSSTCam-imSim’}
[2.2i/runs/DP0.1]

And I’ll try running with the original w_2021_33

I get the same error with w_2021_33.
When I log into the notebooks it’s taking a while. This morning I got the following messages before I got into the Notebooks:
2022-02-17T19:21:34Z [Warning] MountVolume.SetUp failed for volume “butler-secret” : secret “butler-secret” not found
Event log
Server requested
2022-02-17T19:21:33.539594Z [Normal] Successfully assigned nublado2-bobabel/nb-bobabel to lsst-kub015
2022-02-17T19:21:34Z [Warning] MountVolume.SetUp failed for volume “butler-secret” : secret “butler-secret” not found

2022-02-17T19:23:36Z [Warning] Unable to attach or mount volumes: unmounted volumes=[butler-secret], unattached volumes=[lab-environment kube-api-access-n6ddn project passwd teststand group repo instrument dask gshadow idds-config butler-secret tmp home shadow datasets scratch]: timed out waiting for the condition
2022-02-17T19:23:47Z [Normal] Pulling image “Docker Hub
2022-02-17T19:23:48Z [Normal] Successfully pulled image “Docker Hub” in 683.354595ms
2022-02-17T19:23:48Z [Normal] Created container notebook
2022-02-17T19:23:49Z [Normal] Started container notebook

Ok, I see what happened: we registered a new skymap in the main skymaps collection, which is chained to 2.2i/runs/DP0.1, and hence it’s no longer true that there is only one skymap value for the collection used to initialize the butler, and as a result there’s no default value for the skymap data ID key. The fix is to add skymap="DC2" to the data ID.

This is going to affect a lot of other users who rely on that defaulting, and it’s a problem I hadn’t anticipated with having all skymaps go into the same RUN collection. I’ll have to think about the best way to fix this.

It works! Thank you Jim!