We got the error message below. We looked for previous comments on this tutorial, and we could not find anything. A post for tutorial 2 that suggested problems of compatibility between the lsst science pipeline and the rc2_subset data was all that we could find. The solution suggested did not work for us.
Can anyone please help us?
Thank you,
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cli/cmd/commands.py”, line 159, in run
pipeline = script.build(**kwargs, show=show)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cli/script/build.py”, line 87, in build
pipeline = f.makePipeline(args)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 493, in makePipeline
pipeline = Pipeline.from_uri(args.pipeline)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/pipe_base/gb2e446f0d1+d32b27b84e/python/lsst/pipe/base/pipeline.py”, line 330, in from_uri
pipeline: Pipeline = cls.fromIR(pipelineIR.PipelineIR.from_uri(uri))
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/pipe_base/gb2e446f0d1+d32b27b84e/python/lsst/pipe/base/pipelineIR.py”, line 881, in from_uri
with loaded_uri.open(“r”) as buffer:
File “/home/valerio/lsstw/lsstsw/miniconda/envs/lsst-scipipe-5.1.0/lib/python3.10/contextlib.py”, line 135, in enter
return next(self.gen)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/resources/g2029d1bf85+0e2c4d26cd/python/lsst/resources/file.py”, line 488, in open
with open(self.ospath, mode=mode, encoding=encoding) as buffer:
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/valerio/lsstw/lsstsw/rc2_subset/pipelines/DRP.yaml’
I do think this is because you have a mismatch in the version of your tutorial and the version of the pipelines you’re running.
Looks like you’re using a very recent version of RC2_subset and an older version of the tutorial. To find the tutorial corresponding to the most recently weekly. please use: The LSST Science Pipelines — LSST Science Pipelines
Thank you for the input. We tried the updated version of the tutorial, but we still got an error message (see below). Any other suggestion would be much appreciated.
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cli/cmd/commands.py”, line 159, in run
pipeline = script.build(**kwargs, show=show)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cli/script/build.py”, line 87, in build
pipeline = f.makePipeline(args)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 493, in makePipeline
pipeline = Pipeline.from_uri(args.pipeline)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/pipe_base/gb2e446f0d1+d32b27b84e/python/lsst/pipe/base/pipeline.py”, line 330, in from_uri
pipeline: Pipeline = cls.fromIR(pipelineIR.PipelineIR.from_uri(uri))
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/pipe_base/gb2e446f0d1+d32b27b84e/python/lsst/pipe/base/pipelineIR.py”, line 881, in from_uri
with loaded_uri.open(“r”) as buffer:
File “/home/valerio/lsstw/lsstsw/miniconda/envs/lsst-scipipe-5.1.0/lib/python3.10/contextlib.py”, line 135, in enter
return next(self.gen)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/resources/g2029d1bf85+0e2c4d26cd/python/lsst/resources/file.py”, line 488, in open
with open(self.ospath, mode=mode, encoding=encoding) as buffer:
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/drp_pipe/gb906b4b701+012d13edcd/pipelines/HSC/DRP-RC2_subset.yaml’
Hi Valerio - We have been finding some issues with the versions of the tutorial not matching up with their respective releases. Here’s what you can try:
Make sure you are using v24.0.0 of the pipelines. See what you have set up by typing eups list lsst_distrib – this will show you all available versions. If the word “current” doesn’t appear next to v24_0_0 in the list, then you will want to explicitly set up that version using setup -t v24_0_0 lsst_distrib.
Once you have set up the v24.0.0 version of the pipelines, you should also make sure you check out the v24 tag of rc2_subset.
After that, the tutorial at this link should work for you (note that it has v24_0_0 in the URL).
Typing eups list -s lsst_distrib will show only the version that has been setup, making it easier to check which version that is. In Jeff’s command’s output, the word setup is more critical than the word current (which is the default version that will be setup if no other is specified).
Thank you for your suggestions. Indeed, we will need to install v24.0.0 on one of our machine. There were problems in doing that in an Ubuntu OS, so we will try to install CentOS. This may take a while, I will come back to this as soon as possible.
This worked fine up to the tutorial 3. For the tutorial 4, we observed the problems described in this posts. We will try again with a machine operating CentOS. Please let me know if you have other questions.
You seem to have the wrong version of the rubin-env environment. v24_0_0 should be built using lsst-scipipe-4.0.5, not 5.0.0. With that fixed, you can also try a binary install by doing export EUPS_PKGROOT=https://eups.lsst.codes/stack/redhat/el7/conda-system/miniconda3-py38_4.9.2-4.0.5
before your eups distrib install command.
We tried using version 24 for both the science pipelines and the rc_subset data, as suggested. Below there is the latest error message, what are we doing wrong? We will try again on a CentOS machine, but this will take some time…
Thank you,
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/cmd/commands.py”, line 130, in run
qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/qgraph.py”, line 187, in qgraph
qgraph = f.makeGraph(pipelineObj, args)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 565, in makeGraph
butler, collections, run = _ButlerFactory.makeButlerAndCollections(args)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 352, in makeButlerAndCollections
butler, inputs, self = cls._makeReadParts(args)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 288, in _makeReadParts
self = cls(butler.registry, args, writeable=False)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 218, in init
self.inputs = tuple(registry.queryCollections(args.input, flattenChains=True)) if args.input else ()
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registries/sql.py”, line 887, in queryCollections
for record in query.iter(
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registry/wildcards.py”, line 673, in iter
yield from self._search.iter(
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registry/wildcards.py”, line 507, in iter
manager.find(name),
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registry/collections/_base.py”, line 448, in find
raise MissingCollectionError(f"No collection with name ‘{name}’ found.")
lsst.daf.butler.registry._exceptions.MissingCollectionError: No collection with name ‘u/valerio/single_frame’ found.
Hi @valeriocarruba, the fact that this Topic never received a marked solution was brought to my attention today. It would be great to hear if the suggestions worked to resolve your issue.
Thank you for your message. Unfortunately, after getting access to the Rubin Science Platform, I no longer worked on this. What we did at the time did not work. Does that answer the question?