Hi,
I’m working for the LSST:UK DAC team and am trying to ingest data into our butler. I’ve setup the database, registered the instrument and the dataset-type and raw-data ingests work.
Ingesting the deep_coadds from .ecsv files unfortunately gives the error:
lsst.daf.butler._exceptions.DataIdValueError: Could not fetch record for dimension skymap via keys {'band': np.str_('i'), 'skymap': np.str_('lsst_cells_v1'), 'tract': 454, 'patch': 19}
which I think I understand, because there is no skymap registered yet.
Running butler register-skymap $REPO -c name='lsst_cells_v1' then creates a dodecaSkyMap with 12 tracts in REPO and in the database.
And the ingest now fails with Could not fetch record for dimension tract via keys since, I guess, tract 454 is a ringSkyMap tract and not a dodecaSkyMap tract.
How can I convince the butler to register the ringSkyMap from the DP1 pickle file in full and not convert it into a dodecaSkyMap?
Thanks in advance for your help.
I finally found a hack from Brian Yanny and Nate Lust in DM-46717 which pointed me to a workable solution: I copied the text from the skyMap.pickle file into a python skyMap.config file, added import lsst.skymap.ringsSkyMap, and commented out the assert statement.
Then butler register-skymap -C skyMap.config $REPO works and creates the ringsSkyMap entries for all tracts and allows the data to be ingested.
Nevertheless I would like to know if there is a possibility to register a ringsSkyMap by using the pickled skyMap file directly, since this might be also useful for registering skyMaps from other surveys.
Currently there is no way to directly register a skymap from a pickle file. This is mostly historical reasons, and the fact that it was expected that there would only be a very small number of skymaps, and they must be inserted by specific individuals with special permissions in the repositories so there was not a lot of time put into the ergonomics of it. Generally there has only been 1-2 people creating skimps and they have kept copies of the original configs used to insert them (The ability to pull them from the skymap object itself was actually just a side effect). Additionally, because SkyMaps are not just regular dataset types, but also are associated with a bunch of dimension records as well, past transfers of datasets involved dumps from one butler database to another, which transferred all the needed dimension records as well.
All this is just to provide a bit of the historical context to why things are like they are now. There is no deep reason to not have an easy way to reconstruct a skymap, and indeed it has come up several times in the past few months. It is on the radar, and when we get a bit of breathing room this is certainly something we can add, though the exact flow might be a bit different from dealing directly with the pickle file.
Thank you for the explanation and the historical context. It just caught me out, that the default setting was the dodecaSkyMap and the butler didn’t have an option to translate the pickle file into a config file. Good luck with everything!
Hello, Forum admin here – @eckhard.sutorius I’ve marked your first reply as the solution for this topic, as you mentioned it was a workable solution. If that’s incorrect, please feel free to unmark it, and post a reply to clarify whether there remain issues with the skymap?