Trouble Running Custom Pipeline

Support
#1

I’m working toward obtaining the most precise astrometric solution possible using the LSST Science Pipelines (version 28.0.1) along with the 3rd gen Data Butler. I prefer to use Python scripts whenever I can.

So far, I’ve written a basic obs package for the telescope I’m using. At first glance, it seems to work—I can create a Butler repository and ingest some frames. However, I haven’t been able to successfully run any pipeline, either via Python scripts or from the command line.

I’m starting with the master bias step, as I figured it would be the simplest (no overscan subtraction, just combining a few frames). My current understanding is:

  1. I need to create a .yaml file that defines how the data should be processed.
  2. Then, I create a task and run it to generate the output.

I’m not sure exactly where things are going wrong. I also haven’t been able to find an appropriate task for master bias creation. I expected something like cpBiasTask under lsst.cp.pipe.cpBias, but nothing like that seems to exist.

If anyone can share guidance, example workflows, or point me to tutorials or documentation related to creating custom pipelines, I would appreciate it. I’ve gone through the official docs and a few scientific papers, but they haven’t been detailed enough to help me move forward.

#2

Thank you, @beata , for your post!

  1. For combined bias creation: the pipeline is defined in cp_pipe for a given instrument. For LSSTCam, for example, the pipeline yaml file is the following:

If you look at that file, notice that it inherits from a base yaml file in the cp_pipe/pipelines/_ingredients directory, cpBiasLSST.yaml. This file defines the series of tasks that go into the creation of a combined bias. For example, the first task runs instrument signature removal (ISR) on individual input biases (and it’s defined in ip_isr), and the second one does the combination of the individual images into a combined bias (and it’s defined in cp_pipe).

A high-level overview of ISR and calibration production can be found in the following paper:

And some examples on how to run the tasks:

  1. Regarding the creation of tasks and pipetasks using the LSST Science Pipelines:

Please let us know if this information is helpful, and feel free to follow up with more questions if needed!