How to share master calibration frames among repos in Gen3

Support
#1

Hi everyone,
We are processing DECam data using the LSST pipeline Gen3. The targets we are studying are scattered on the sky, and their exposures span years. We process the science exposures of each target in different repos. In Gen3 we need to stack the raw calibration frames to build master calibration frames (e.g. flat, bias). However, we are wondering if the master calibration frames built in the LSST pipeline can be shared among the repos, so that we don’t need to remake the master calibration frames for each target in each repo. This will save us a lot of time and disk space. Could anyone give us some guidance? Any suggestions will be appreciated.

#2 
$ butler export-calibs --help     
Usage: butler export-calibs [OPTIONS] REPO DIRECTORY [COLLECTIONS] ...

  Export calibrations from the butler for import elsewhere.

  REPO is the URI or path to an existing data repository root or configuration
  file.

  DIRECTORY is the folder to receive the exported calibrations.

  COLLECTIONS are the collection to export calibrations from.

Options:
  -d, --dataset-type TEXT ...     Specific DatasetType(s) to export.
  -t, --transfer [auto|link|symlink|hardlink|copy|move|relsymlink|direct]
                                  The external data transfer mode.
  -h, --help                      Show this message and exit.

is the mechanism we have.

3 Likes
#3

Thank you very much. Does DIRECTORY have to be another repo?

#4

No, DIRECTORY is a place for you to write the calibration files. The idea being you would then transfer that directory content to some other place so you can then run butler import for the target repo.

1 Like