Generation 3 Butler notes using S3


Contents



butler Command line tasks


butler Commands help

Creating a generation 3 butler repository




Adding an instrument to the GEN 3 butler repository




Ingest raw frames into from a directory into the butler registry




Convert a Butler gen 2 repository into a gen 3 repository



Importing data accross two GEN 3 repositories




Butler using a jupyter notebook


LSST Stack Imports


Accessing the butler repository


Accessing the data registry

The registry is a good tool for investigating a repo (more on the registry schema can be found here). For example, we can get a list of all collections, which includes the HSC/raw/all collection that we were using before

now that we "know" that HSC/raw/all exists, let's create our butler with this collection:

We can also use the registry to get a list of all dataset types

We suspect that this is all datasetTypes that the processing has tried to create during the processing. There may be intermediate products that were created during processing, but no longer exist.

It is now possible to get all DatasetRef (including dataId) for a specific datasetType in a specific collection with a query like the one that follows

Ok, now that we know what collections exist (HSC/raw/all in particular), the datasetTypes that are defined for that collection, and the datasetRefs (which contain dataIds) for data products of the requested type. This is all the information that we need to get the dataset of interest.

From the list above, I choose index 16 and with this we will find the dataId

DatasetRef is a combination of dataset type and dataId and can refer to an explicit dataset in a specific run (if ref.dataId is defined)


Plotting

How to create a table using the data

More plotting


Getting the URIs


Exporting


Importing

Check if the files transfered