Generation 3 Butler notes using S3


butler Command line tasks

butler Commands help

Creating a generation 3 butler repository

Adding an instrument to the GEN 3 butler repository

Ingest raw frames into from a directory into the butler registry

Defining the visits system in a butler repository

Convert a Butler gen 2 repository into a gen 3 repository

Importing data accross two GEN 3 repositories

Butler using a jupyter notebook

LSST Stack Imports

Accessing the butler repository

Accessing the data registry

The registry is a good tool for investigating a repo (more on the registry schema can be found here). For example, we can get a list of all collections, which includes the HSC/raw/all collection that we were using before

now that we "know" that HSC/raw/all exists, let's create our butler with this collection:

We can also use the registry to get a list of all dataset types

We suspect that this is all datasetTypes that the processing has tried to create during the processing. There may be intermediate products that were created during processing, but no longer exist.

It is now possible to get all DatasetRef (including dataId) for a specific datasetType in a specific collection with a query like the one that follows

Ok, now that we know what collections exist (HSC/raw/all in particular), the datasetTypes that are defined for that collection, and the datasetRefs (which contain dataIds) for data products of the requested type. This is all the information that we need to get the dataset of interest.

From the list above, I choose index 16 and with this we will find the dataId

DatasetRef is a combination of dataset type and dataId and can refer to an explicit dataset in a specific run (if ref.dataId is defined)


How to create a table using the data

More plotting