Hi, I have started using LSST stack recently and am running through the Getting started tutorial 2. When I try to run the ‘Running single frame processing’ commands using pipetask run, it gives me following error: FileNotFoundError: [Errno 2] No such file or directory: '/jbodstorage/data_ms19054/lsst_stack/rc2_subset/$DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml'. Attaching the screenshot of the terminal below for more details.
I cannot locate the file DRP-RC2.yamlanywhere on my server. Running echo $DRP_PIPE_DIR also gives no result. Would be very helpful if someone can help me with understanding what is wrong.
Allegedly the tutorial was written for w_2021_34 and v23 was branched from w_2021_40. I have not tried v23 but I have tried w_2022_14 and the pipetask command works fine for me.
I think this error comes from the drp_pipe package not being setup. If an environment variable is not defined then its value is not replaced in a string. The drp_pipe package did not get included in the science pipelines until w_2022_06 so if you are using v23 there should be no mention of DRP_PIPE_DIR. Can you search through your pipelines tree to look for mentions of DRP_PIPE_DIR? Is it possible you have a split stack setup with some packages from v23 and others from a more recent weekly?
My apologies. It seems that the rc2_subset repo was updated 2 months ago to use the DRP_PIPE definition of the pipeline.
Unfortunately, because the package is not part of the release process it does not get given weekly tags. The v23 release was made at the end of September. Can you please checkout commit 8208b4c of rc2_subset since that was made roughly around that time. Commit 3095d1e seems to be the closest to v23.
Using $OBS_SUBARU_DIR/pipelines/DRP.yaml directly in the DRP.yaml file did not work, as it was giving few more errors related to some elements being not found.
To resolve that, I checked the commit 8208b4c of rc2_subset as suggested by @timj. Found the DRP.yaml file before the change of adding DRP_PIPE_DIR variable was made and used that version of the repo to run the same commands. It works successfully.
