Quantum-backed butlers and dataset merging

Support
#1

Hi all,

I have been trying to use the quantum-backed butler to speed up the processing of some HSC data, and am struggling with the whole “merging” of the datasets into the actual butler after execution has finished. The runs appear in the butler directory, but the collections and datasets are never registered in the butler. I am certain my confusion comes from a lack of understanding about how quantum-backed butlers and SQL databases work, and I was hoping someone could provide an explanation about how these things actually work. Is there a command to run after pipetask run-qbb to merge the datasets into the butler/SQL database? For reference I am running v26 of the pipeline.

#2

Yes

$ butler --log-level=VERBOSE --long-log transfer-from-graph --register-dataset-types --update-output-chain $GRAPH  $REPO

where $GRAPH is the quantum graph you are using and $REPO is your target butler.

1 Like
#3

Thanks, Tim! @ess20clsi I’m going to mark Tim’s response as the solution.

1 Like