Creating bias frames for each detector

Support
#1

I’m trying to construct a bias frame and I would like to filter it by detector. I am trying to run it through a for loop to process each detector individually. I have gotten this far:

pipetask run --register-dataset-types \
-b $BUTLER_DIR \
-i HSC/raw/all,HSC/calib \
-o u/NH/bias-construction \
-p $CP_PIPE_DIR/pipelines/HyperSuprimeCam/cpBias.yaml \
-d "instrument='HSC' AND exposure.observation_type='bias'"

I am planning to loop through this command for every detector. I am just unsure how to filter by a specific detector. Does anyone have any insight on how I might accomplish this?

#2

Have you tried adding something like

AND detector=41

within the -d argument? There’s an example like this (though not specifically for bias creation) in the pipelines.lsst.io Getting Started tutorial, which also uses HSC:

https://pipelines.lsst.io/getting-started/singleframe.html

Thanks…

#3

Is there a reason you aren’t letting the pipeline system automatically iterate over every detector? You would get the same answer and if you run with multiple cores, -j N, it will be more efficient (and if you have a batch system you can use BPS to make it even faster).

#4

We were planning on running multiple headless jobs to make the process go faster. Unfortunately the way cores are allocated on the system we use makes it really difficult to do it with -j N.

#5

Thanks, this seemed to work!

#6

If you ever do end up with a batch system like slurm or htcondor we have software that can make this all run efficiently on multiple nodes. We use it all the time for data release processing.