Diagram describing the inputs and outputs of tasks

Support
#1

Hi all,

is there some kind of diagram or document that explains what are the intputs and outputs of any task (say ProcessCCD, makeCoaddTempExp, assembleCoadd etc.), or even better would represent all the tasks from raw to, say, image difference catalogs.
Basically something that would link most or all the entries of https://github.com/lsst/obs_subaru/blob/master/policy/HscMapper.yaml ?

Best,

Ben

PS: Ideally I want to have that diagram for Gen2 tasks.

2 Likes
#2

For Gen2-only Tasks, there’s no real solution other than to read the code (and there’s no guarantee that this information will be particularly localized in the code).

But most of the Tasks used in DRP processing, including all of the ones you mentioned (and I think most of the ones in AP processing) are now hybrids that work in both Gen2 and Gen3, and you can use the Gen3 machinery (the pipetask tool from ctrl_mpexec) to generate diagrams like this:

pipetask build -p $PIPE_TASKS_DIR/pipelines/ProcessCcd.yaml --pipeline-dot ProcessCcd.dot
dot ProcessCcd.dot -T svg > ProcessCcd.svg

You can find several other YAML pipeline descriptions in that directory, and you can build your own and modify an existing one using various other options to pipetask build.

In some cases, the details of the inputs and outputs (mostly calibrations used by ISR) depend on the configuration, so you may want to pass something like --instrument lsst.obs.subaru.HyperSuprimeCam to pick up the overrides for the instrument you want to use.

The dot command here is not provided with the stack, but it’s available on most OSs as part of the graphviz package. There are many different output formats, but whatever you do, be prepared to zoom in and scroll around - the graphs are often quite large with small text, because there’s a lot to display.

Finally, you’ll note that while most Tasks are exactly the same as Gen2, because we use the exact same Task in both places, a few are either missing because they don’t have Gen3 versions yet (FGCM and jointcal) or have a completely different Gen3 Task (MakeCoaddTempExp -> MakeWarpTask may be the only one in the default pipelines). These usually reflect a slightly different granularity of processing, but the inputs and outputs are the same.

2 Likes