Centrally-defined butler datasets

mapper
obspackages
dm-dev
butler
Tags: #<Tag:0x00007fd88e5d3c20> #<Tag:0x00007fd88e5d3ab8> #<Tag:0x00007fd88e5d3748> #<Tag:0x00007fd88e5d3568>

(Paul Price) #1

As part of my work on DM-3372 (“stack provenance”), I added a new feature to the CameraMapper that allows us to provide a default specification for datasets. This allows us to add datasets that will be defined in the same way for all cameras. This meant that I didn’t have to add a packages dataset in all the obs_* packages (to save the package version information), but by putting it in a single place it takes effect for all the obs_* packages. Beyond that convenience, I hope it will also allow us to simplify the existing obs_* packages by moving most of the coadd products into a single place.

The default dataset specifications are stored in daf_butlerUtils/policy as images.yaml, exposures.yaml, calibrations.yaml and datasets.yaml, according to which category of dataset it is. They have the same information as what would go in our *Mapper.paf files, but (as the file extensions suggest) are in YAML. Files that don’t currently exist are treated as empty.


DM Highlights 2016-06-01 to 2016-06-07
(Nate Pease) #2

@price, why did you break the different categories into separate yaml files, instead of having one yaml file, named something like “defaultPolicy.yaml”?


(Paul Price) #3

It was my first foray into YAML, and it seemed dividing the namespace using files was neater and/or easier than having multiple namespaces in a single file (e.g., all elements in a file have the same format). You’re welcome to refactor it if you want.


(Nate Pease) #4

understand. thanks.

(this post must be at least 20 characters)