Hi, I have started using LSST stack recently and am running through the Getting started tutorial 2. When I try to run the ‘Running single frame processing’ commands using
pipetask run, it gives me following error:
FileNotFoundError: [Errno 2] No such file or directory: '/jbodstorage/data_ms19054/lsst_stack/rc2_subset/$DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml'. Attaching the screenshot of the terminal below for more details.
I cannot locate the file
DRP-RC2.yamlanywhere on my server. Running
echo $DRP_PIPE_DIR also gives no result. Would be very helpful if someone can help me with understanding what is wrong.
Details of the error:
Hi @prajakta1-618 . Can you output
eups list lsst_distrib
My guess is you are using the v_23_0_x pipeline. The move of the pipeline yaml files to DRP_PIPE_DIR happened after this release. You can try using:
The other possibility is to install the latest weekly:
eups distrib install lsst_distrib -t w_2022_14
setup -v lsst_distrib -t w_2022_14
This should then setup the $DRP_PIPE_DIR variable.
Allegedly the tutorial was written for w_2021_34 and v23 was branched from w_2021_40. I have not tried v23 but I have tried w_2022_14 and the
pipetask command works fine for me.
I think this error comes from the drp_pipe package not being setup. If an environment variable is not defined then its value is not replaced in a string. The
drp_pipe package did not get included in the science pipelines until w_2022_06 so if you are using v23 there should be no mention of DRP_PIPE_DIR. Can you search through your pipelines tree to look for mentions of DRP_PIPE_DIR? Is it possible you have a split stack setup with some packages from v23 and others from a more recent weekly?
My apologies. It seems that the
rc2_subset repo was updated 2 months ago to use the DRP_PIPE definition of the pipeline.
Unfortunately, because the package is not part of the release process it does not get given weekly tags. The v23 release was made at the end of September. Can you please checkout commit
rc2_subset since that was made roughly around that time. Commit
3095d1e seems to be the closest to v23.
$OBS_SUBARU_DIR/pipelines/DRP.yaml directly in the
DRP.yaml file did not work, as it was giving few more errors related to some elements being not found.
To resolve that, I checked the commit
rc2_subset as suggested by @timj. Found the
DRP.yaml file before the change of adding DRP_PIPE_DIR variable was made and used that version of the repo to run the same commands. It works successfully.
Thank you very much @surhudm and @timj!