Generation 3 Butler notes using S3

---

Contents

• Useful links

• butler Command line tasks

  • Creating a generation 3 butler repository
  • Adding an instrument to the GEN 3 butler repository
  • Ingest raw frames into from a directory into the butler registry
  • Importing data accross two GEN 3 repositories
  • Convert a Butler gen 2 repository into a gen 3 repository

• Butler using a jupyter notebook

  • LSST Stack Imports
  • Accessing the butler repository
  • Accessing the data registry
  • Plotting
  • Getting the URIs
  • Exporting
  • Importing

---

Useful links

• https://pipelines.lsst.io/modules/lsst.daf.butler/scripts/butler.py.html
• https://github.com/LSSTScienceCollaborations/StackClub/blob/master/Basics/Gen3ButlerTutorial.ipynb
• https://dmtn-073.lsst.io/
• https://stackclub.readthedocs.io/en/latest/notebooks.html#getting-started
• https://docushare.lsstcorp.org/docushare/dsweb/Get/presentation-890/Gen3%20Middleware.pdf
• https://pipelines.lsst.io/modules/lsst.ap.pipe/getting-started.html?highlight=gen
• https://pipelines.lsst.io/search.html?q=gen+3&check_keywords=yes&area=default
• https://pipelines.lsst.io/v/weekly/py-api/lsst.daf.butler.Butler.html
• https://github.com/LSSTScienceCollaborations/StackClub
• https://notebook.community/hsinfang/lsst-notes/gen3butler/gen3notes
• https://github.com/lsst/testdata_jointcal/tree/master/scripts

---

butler Command line tasks

---

Creating a generation 3 butler repository

• Step 1 : Creating Butler.yaml file
  
  

  • Firstly, we will need to define the type of storage we will be using e.g ChainedDatastore,InMemoryDatastore,FileDatastore and WebdavDatastore(https://pipelines.lsst.io/modules/lsst.daf.butler/index.html, Crtl-F Example datastores) in this case I'll be using Filedatastore which is the class for storage using s3.
    
    

  • We will also need to create a SQL file for the registry (e.g vi test.sqlite3)
    
    

  • In my case I had to create an S3 bucket on echo which will be the butler repository. I used Rclone (https://rclone.org/docs/) to do this "rclone mkdir remote:bucket_name"
    
    

  • Now we have all of this, create a new file and call it butler.yaml. Within that file you have the class of the data storage which is found on the link above and the path to the SQL file. Example
    
    

    • datastore:
      

        cls: lsst.daf.butler.datastores.fileDatastore.FileDatastore 
        root: s3://bucket_name 
      

      registry:
      

        db: sqlite:////home/test.sqlite3 

• Step 2 : Configuring the butler repository
  
  

  • Now we have got the butler.yaml file we can create an empty Gen3 Butler repository.
    
    

  • We do this by using "butler create" which is a command line task (https://pipelines.lsst.io/modules/lsst.daf.butler/scripts/butler.py.html)
    
    

  • We run:
    
    

    • » butler create s3://bucket_name --seed-config butler.yaml --override
      
      
    • where s3://bucket_name is the REPO which is the URI or path to the new repository
      
      

  • Now we have created a generation 3 Butler repository, if we check our s3 butler repository will see that the butler.yaml is now in the repository. We can also check the sql file and see all the tables loaded in which help query the dataset
    
    

---

Adding an instrument to the GEN 3 butler repository

• Step 1: Find the instrument class
  
  

  • For lsst use this link (https://pipelines.lsst.io/modules/lsst.obs.lsst/index.html, Ctrl-F lsst.obs.lsst Package) an example of class is "lsst.obs.lsst.LsstComCam"
    
    

• Step 2: Running the "register-instrument" Command
  
  

  • » butler register-instrument s3://bucket_name lsst.obs.subaru.HyperSuprimeCam
    
    

  • where s3://bucket_name is the REPO which is the URI or path to the new repository and lsst.obs.subaru.HyperSuprimeCam is the instrument class

---

Ingest raw frames into from a directory into the butler registry

• Step 1: Adding an instrument to the GEN 3 butler repository
  
  

  • Make sure that an instrument has been added the into the GEN 3 butler repository (Look the above for instructions to how to add an instrument to the butler repsitory)
    
    

• Step 2: Running the "ingest-raws" Command
  
  

  • » butler ingest-raws s3://bucket_name /home/lsst_stack/testdata_ci_hsc/raw
    
    

  • where s3://bucket_name is the REPO which is the URI or path to the new repository and /home/lsst_stack/testdata_ci_hsc/raw is the LOCATIONs specifies files to ingest and/or locations to search for files.

---

Convert a Butler gen 2 repository into a gen 3 repository

• Step 1: Set up a Gen 3 butler repository(Look above for instructions)
  
  

• Step 2: Copy the gen 2 _mapper file into the gen 3 butler repository
  
  

  • » For S3 use rclone copy remote:bucket_name
    
    

• Step 3: Running the "convert" Command
  
  

  • » butler convert s3://bucket_name --gen2root /home/lsst_stack/DATA
    
    

  • where s3://bucket_name is the REPO which is the URI or path to the new repository, --gen2root Root path of the gen 2 repo to be converted and/home/lsst_stack/DATA is the path to the gen 2 repo.
    
    

  • The tutorial for create a gen 2 repository is here https://pipelines.lsst.io/getting-started/data-setup.html
    
    

---

Importing data accross two GEN 3 repositories

• Step 1: Export the data
  
  

  • Firstly you will have export data from the repository with the data currently in it.(how to export data)
    
    

• Step 2: Running the "ingest-raws" Command
  
  

  • » butler import s3://bucket_name_new s3://bucket_name --export-file exports.yaml
    
    
  • where s3://bucket_name is the REPO which is the URI or path to the repository with the data and s3://bucket_name_new is the REPO which is the URI or path to the repository ehre you want to put your data .

Butler using a jupyter notebook

---

LSST Stack Imports

• butler class (https://pipelines.lsst.io/v/weekly/py-api/lsst.daf.butler.Butler.html)
  
• Image display class(https://pipelines.lsst.io/v/weekly/modules/lsst.afw.display/index.html)
  
• pylab for plotting
  

import lsst.daf.butler as dafButler
import lsst.afw.display as afwDisplay
import pylab as plt
import os,glob
import lsst.geom as geom

---

Accessing the butler repository

butler = dafButler.Butler("s3://joshuakitenge-DATA")

---

Accessing the data registry

registry = butler.registry
registry.isWriteable()

# We can examine the registry with
#help(registry)

()

The registry is a good tool for investigating a repo (more on the registry schema can be found here). For example, we can get a list of all collections, which includes the HSC/raw/all collection that we were using before

for c in registry.queryCollections():
    print(c)

HSC/raw/all
HSC/calib
HSC/calib/unbounded
HSC/calib/curated/1970-01-01T00:00:00
HSC/calib/curated/2013-01-31T00:00:00
HSC/calib/curated/2014-04-03T00:00:00
HSC/calib/curated/2014-06-01T00:00:00
HSC/calib/curated/2015-11-06T00:00:00
HSC/calib/curated/2016-04-01T00:00:00
HSC/calib/curated/2016-11-22T00:00:00
HSC/calib/curated/2016-12-23T00:00:00

now that we "know" that HSC/raw/all exists, let's create our butler with this collection:

butler = dafButler.Butler("s3://joshuakitenge-DATA",collections='HSC/raw/all')
registry = butler.registry

We can also use the registry to get a list of all dataset types

for x in registry.queryDatasetTypes():
    print(x)

DatasetType('raw', {band, instrument, detector, physical_filter, exposure}, Exposure)
DatasetType('camera', {instrument}, Camera, isCalibration=True)
DatasetType('defects', {instrument, detector}, Defects, isCalibration=True)
DatasetType('bfKernel', {instrument}, NumpyArray, isCalibration=True)
DatasetType('transmission_optics', {instrument}, TransmissionCurve, isCalibration=True)
DatasetType('transmission_sensor', {instrument, detector}, TransmissionCurve, isCalibration=True)
DatasetType('transmission_filter', {band, instrument, physical_filter}, TransmissionCurve, isCalibration=True)
DatasetType('transmission_atmosphere', {instrument}, TransmissionCurve, isCalibration=True)

We suspect that this is all datasetTypes that the processing has tried to create during the processing. There may be intermediate products that were created during processing, but no longer exist.

It is now possible to get all DatasetRef (including dataId) for a specific datasetType in a specific collection with a query like the one that follows

datasetRefs = list(registry.queryDatasets(datasetType='raw',collections=['HSC/raw/all']))
for ref in datasetRefs:
    print(ref.dataId)

{band: r, instrument: HSC, detector: 11, physical_filter: HSC-R, exposure: 903344}
{band: r, instrument: HSC, detector: 5, physical_filter: HSC-R, exposure: 903344}
{band: r, instrument: HSC, detector: 0, physical_filter: HSC-R, exposure: 903344}
{band: i, instrument: HSC, detector: 25, physical_filter: HSC-I, exposure: 903990}
{band: i, instrument: HSC, detector: 18, physical_filter: HSC-I, exposure: 903990}
{band: r, instrument: HSC, detector: 6, physical_filter: HSC-R, exposure: 903346}
{band: r, instrument: HSC, detector: 1, physical_filter: HSC-R, exposure: 903346}
{band: r, instrument: HSC, detector: 12, physical_filter: HSC-R, exposure: 903346}
{band: i, instrument: HSC, detector: 23, physical_filter: HSC-I, exposure: 903986}
{band: i, instrument: HSC, detector: 22, physical_filter: HSC-I, exposure: 903986}
{band: i, instrument: HSC, detector: 100, physical_filter: HSC-I, exposure: 903986}
{band: i, instrument: HSC, detector: 16, physical_filter: HSC-I, exposure: 903986}
{band: i, instrument: HSC, detector: 17, physical_filter: HSC-I, exposure: 903988}
{band: i, instrument: HSC, detector: 23, physical_filter: HSC-I, exposure: 903988}
{band: i, instrument: HSC, detector: 24, physical_filter: HSC-I, exposure: 903988}
{band: i, instrument: HSC, detector: 16, physical_filter: HSC-I, exposure: 903988}
{band: r, instrument: HSC, detector: 10, physical_filter: HSC-R, exposure: 903342}
{band: r, instrument: HSC, detector: 4, physical_filter: HSC-R, exposure: 903342}
{band: r, instrument: HSC, detector: 100, physical_filter: HSC-R, exposure: 903342}
{band: i, instrument: HSC, detector: 4, physical_filter: HSC-I, exposure: 904010}
{band: i, instrument: HSC, detector: 10, physical_filter: HSC-I, exposure: 904010}
{band: i, instrument: HSC, detector: 100, physical_filter: HSC-I, exposure: 904010}
{band: r, instrument: HSC, detector: 23, physical_filter: HSC-R, exposure: 903334}
{band: r, instrument: HSC, detector: 16, physical_filter: HSC-R, exposure: 903334}
{band: r, instrument: HSC, detector: 100, physical_filter: HSC-R, exposure: 903334}
{band: r, instrument: HSC, detector: 22, physical_filter: HSC-R, exposure: 903334}
{band: i, instrument: HSC, detector: 6, physical_filter: HSC-I, exposure: 904014}
{band: i, instrument: HSC, detector: 12, physical_filter: HSC-I, exposure: 904014}
{band: i, instrument: HSC, detector: 1, physical_filter: HSC-I, exposure: 904014}
{band: r, instrument: HSC, detector: 18, physical_filter: HSC-R, exposure: 903338}
{band: r, instrument: HSC, detector: 25, physical_filter: HSC-R, exposure: 903338}
{band: r, instrument: HSC, detector: 24, physical_filter: HSC-R, exposure: 903336}
{band: r, instrument: HSC, detector: 17, physical_filter: HSC-R, exposure: 903336}

Ok, now that we know what collections exist (HSC/raw/all in particular), the datasetTypes that are defined for that collection, and the datasetRefs (which contain dataIds) for data products of the requested type. This is all the information that we need to get the dataset of interest.

From the list above, I choose index 16 and with this we will find the dataId

ref = datasetRefs[16]
print(ref.dataId)

{band: r, instrument: HSC, detector: 10, physical_filter: HSC-R, exposure: 903342}

---

Plotting

• Using the dataset from above we can get the image (more on plotting in
  https://github.com/LSSTScienceCollaborations/StackClub/blob/master/Basics/Gen3ButlerTutorial.ipynb
  https://github.com/LSSTScienceCollaborations/StackClub/blob/master/Basics/Calexp_guided_tour.ipynb)
  
• get(datasetRefOrType, …): Retrieve a stored dataset.

# To get the image,we pass the dataId
test = butler.get(ref)

#And plot!
afwDisplay.setDefaultBackend('matplotlib') 
fig = plt.figure(figsize=(10,8))
afw_display = afwDisplay.Display(1)
afw_display.scale('asinh', 'zscale')
afw_display.mtv(test)
plt.gca().axis('off')

(-0.5, 2143.5, -0.5, 4240.5)

---

Getting the URIs

• Still unclear about sue case of getURIs ?
  
• Getting the URI of the raw data
  
• getURI(datasetRefOrType, …): Return the URI to the Dataset
  
• getURIs(datasetRefOrType, …): Returns the URIs associated with the dataset
  
• there must be a more effiient way of doing this

Raw_URI=[]
Raw_URIs=[]
for ref in butler.registry.queryDatasets("raw", collections=['HSC/raw/all']):#, where="detector = 22"):
    uri = butler.getURI(ref)
    Raw_URI.append(uri)
    print("{}\n".format(uri))
   
    
for ref in butler.registry.queryDatasets("raw", collections=['HSC/raw/all']):#, where="detector = 22"):
    uri = butler.getURIs(ref)
    print("{}\n".format(uri))
          
#print("{}\n".format(Raw_URI))
#print("{}\n".format(Raw_URIs))

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903344_11_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903344_5_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903344_0_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903990_25_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903990_18_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903346_6_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903346_1_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903346_12_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903986_23_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903986_22_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903986_100_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903986_16_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903988_17_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903988_23_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903988_24_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903988_16_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903342_10_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903342_4_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903342_100_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904010_4_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904010_10_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904010_100_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903334_23_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903334_16_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903334_100_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903334_22_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904014_6_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904014_12_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904014_1_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903338_18_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903338_25_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903336_24_HSC_HSC_raw_all.fits

s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903336_17_HSC_HSC_raw_all.fits

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903344_11_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903344_5_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903344_0_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903990_25_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903990_18_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903346_6_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903346_1_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903346_12_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903986_23_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903986_22_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903986_100_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903986_16_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903988_17_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903988_23_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903988_24_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_903988_16_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903342_10_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903342_4_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903342_100_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904010_4_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904010_10_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904010_100_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903334_23_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903334_16_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903334_100_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903334_22_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904014_6_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904014_12_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/i/HSC-I/raw_i_HSC-I_904014_1_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903338_18_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903338_25_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903336_24_HSC_HSC_raw_all.fits"), {})

(ButlerURI("s3://joshuakitenge-DATA/HSC/raw/all/raw/r/HSC-R/raw_r_HSC-R_903336_17_HSC_HSC_raw_all.fits"), {})

---

Exporting

• useful link https://github.com/lsst/testdata_jointcal/blob/master/scripts/export_gen3_hsc.py
• Exporting all the raw files

with butler.export(filename = "/home/vrs42921/lsst_stack/exports.yaml") as export:
    export.saveDatasets(butler.registry.queryDatasets("raw", collections=['HSC/raw/all']))

---

Importing

• Using either a preexisting GEN 3 butler repository or a a new GEN 3 butler repository
  
• Set up the but client for the new repository and make sure that "writeable=True" is there
  
• As I'm transfer the data from one repository to another I'll set ""transfer="copy" otherwise transfer has Options:auto, link , symlink , hardlink , copy , move , relsymlink and direct

butler_im = dafButler.Butler("s3://kitenge_test_2",writeable=True)
butler_im.import_(directory="s3://joshuakitenge-DATA",filename="exports.yaml",transfer="copy")

True

Check if the files transfered

butler_im_test = dafButler.Butler("s3://kitenge_test_2")
reg =butler_im_test.registry
for c in reg.queryCollections():
    print(c)

HSC/raw/all

datasetRefs_test = list(reg.queryDatasets(datasetType='raw',collections=['HSC/raw/all']))
for ref in datasetRefs_test:
    print(ref.dataId)

{band: i, instrument: HSC, detector: 100, physical_filter: HSC-I, exposure: 903986}
{band: i, instrument: HSC, detector: 16, physical_filter: HSC-I, exposure: 903986}
{band: i, instrument: HSC, detector: 22, physical_filter: HSC-I, exposure: 903986}
{band: i, instrument: HSC, detector: 23, physical_filter: HSC-I, exposure: 903986}
{band: i, instrument: HSC, detector: 16, physical_filter: HSC-I, exposure: 903988}
{band: i, instrument: HSC, detector: 17, physical_filter: HSC-I, exposure: 903988}
{band: i, instrument: HSC, detector: 23, physical_filter: HSC-I, exposure: 903988}
{band: i, instrument: HSC, detector: 24, physical_filter: HSC-I, exposure: 903988}
{band: i, instrument: HSC, detector: 18, physical_filter: HSC-I, exposure: 903990}
{band: i, instrument: HSC, detector: 25, physical_filter: HSC-I, exposure: 903990}
{band: i, instrument: HSC, detector: 100, physical_filter: HSC-I, exposure: 904010}
{band: i, instrument: HSC, detector: 10, physical_filter: HSC-I, exposure: 904010}
{band: i, instrument: HSC, detector: 4, physical_filter: HSC-I, exposure: 904010}
{band: i, instrument: HSC, detector: 12, physical_filter: HSC-I, exposure: 904014}
{band: i, instrument: HSC, detector: 1, physical_filter: HSC-I, exposure: 904014}
{band: i, instrument: HSC, detector: 6, physical_filter: HSC-I, exposure: 904014}
{band: r, instrument: HSC, detector: 100, physical_filter: HSC-R, exposure: 903334}
{band: r, instrument: HSC, detector: 16, physical_filter: HSC-R, exposure: 903334}
{band: r, instrument: HSC, detector: 22, physical_filter: HSC-R, exposure: 903334}
{band: r, instrument: HSC, detector: 23, physical_filter: HSC-R, exposure: 903334}
{band: r, instrument: HSC, detector: 17, physical_filter: HSC-R, exposure: 903336}
{band: r, instrument: HSC, detector: 24, physical_filter: HSC-R, exposure: 903336}
{band: r, instrument: HSC, detector: 18, physical_filter: HSC-R, exposure: 903338}
{band: r, instrument: HSC, detector: 25, physical_filter: HSC-R, exposure: 903338}
{band: r, instrument: HSC, detector: 100, physical_filter: HSC-R, exposure: 903342}
{band: r, instrument: HSC, detector: 10, physical_filter: HSC-R, exposure: 903342}
{band: r, instrument: HSC, detector: 4, physical_filter: HSC-R, exposure: 903342}
{band: r, instrument: HSC, detector: 0, physical_filter: HSC-R, exposure: 903344}
{band: r, instrument: HSC, detector: 11, physical_filter: HSC-R, exposure: 903344}
{band: r, instrument: HSC, detector: 5, physical_filter: HSC-R, exposure: 903344}
{band: r, instrument: HSC, detector: 12, physical_filter: HSC-R, exposure: 903346}
{band: r, instrument: HSC, detector: 1, physical_filter: HSC-R, exposure: 903346}
{band: r, instrument: HSC, detector: 6, physical_filter: HSC-R, exposure: 903346}