Problems with tutorial 4 of the LSST Science Pipelines: Global Caibration

Support
#1

Dear Sir or Madam,

We are having trouble with the tutorial 4 of the LSST Science Pipelines. While we run tutorial 2 perfectly, when we tried the command:

pipetask run -b $RC2_SUBSET_DIR/SMALL_HSC/butler.yaml
-p $RC2_SUBSET_DIR/pipelines/DRP.yaml#fgcm
-i u/$USER/single_frame
-o u/$USER/fgcm
–register-dataset-types

We got the error message below. We looked for previous comments on this tutorial, and we could not find anything. A post for tutorial 2 that suggested problems of compatibility between the lsst science pipeline and the rc2_subset data was all that we could find. The solution suggested did not work for us.

Can anyone please help us?

Thank you,

lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cli/cmd/commands.py”, line 159, in run
pipeline = script.build(**kwargs, show=show)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cli/script/build.py”, line 87, in build
pipeline = f.makePipeline(args)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 493, in makePipeline
pipeline = Pipeline.from_uri(args.pipeline)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/pipe_base/gb2e446f0d1+d32b27b84e/python/lsst/pipe/base/pipeline.py”, line 330, in from_uri
pipeline: Pipeline = cls.fromIR(pipelineIR.PipelineIR.from_uri(uri))
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/pipe_base/gb2e446f0d1+d32b27b84e/python/lsst/pipe/base/pipelineIR.py”, line 881, in from_uri
with loaded_uri.open(“r”) as buffer:
File “/home/valerio/lsstw/lsstsw/miniconda/envs/lsst-scipipe-5.1.0/lib/python3.10/contextlib.py”, line 135, in enter
return next(self.gen)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/resources/g2029d1bf85+0e2c4d26cd/python/lsst/resources/file.py”, line 488, in open
with open(self.ospath, mode=mode, encoding=encoding) as buffer:
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/valerio/lsstw/lsstsw/rc2_subset/pipelines/DRP.yaml’

#2

I do think this is because you have a mismatch in the version of your tutorial and the version of the pipelines you’re running.

Looks like you’re using a very recent version of RC2_subset and an older version of the tutorial. To find the tutorial corresponding to the most recently weekly. please use: The LSST Science Pipelines — LSST Science Pipelines

#3

Thank you for the input. We tried the updated version of the tutorial, but we still got an error message (see below). Any other suggestion would be much appreciated.

lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cli/cmd/commands.py”, line 159, in run
pipeline = script.build(**kwargs, show=show)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cli/script/build.py”, line 87, in build
pipeline = f.makePipeline(args)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/ctrl_mpexec/g89ce1456cb+8822c702cc/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 493, in makePipeline
pipeline = Pipeline.from_uri(args.pipeline)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/pipe_base/gb2e446f0d1+d32b27b84e/python/lsst/pipe/base/pipeline.py”, line 330, in from_uri
pipeline: Pipeline = cls.fromIR(pipelineIR.PipelineIR.from_uri(uri))
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/pipe_base/gb2e446f0d1+d32b27b84e/python/lsst/pipe/base/pipelineIR.py”, line 881, in from_uri
with loaded_uri.open(“r”) as buffer:
File “/home/valerio/lsstw/lsstsw/miniconda/envs/lsst-scipipe-5.1.0/lib/python3.10/contextlib.py”, line 135, in enter
return next(self.gen)
File “/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/resources/g2029d1bf85+0e2c4d26cd/python/lsst/resources/file.py”, line 488, in open
with open(self.ospath, mode=mode, encoding=encoding) as buffer:
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/valerio/lsstw/lsstsw/stack/lsst-scipipe-5.1.0/Linux64/drp_pipe/gb906b4b701+012d13edcd/pipelines/HSC/DRP-RC2_subset.yaml’

#4

Hi Valerio - We have been finding some issues with the versions of the tutorial not matching up with their respective releases. Here’s what you can try:

Make sure you are using v24.0.0 of the pipelines. See what you have set up by typing eups list lsst_distrib – this will show you all available versions. If the word “current” doesn’t appear next to v24_0_0 in the list, then you will want to explicitly set up that version using setup -t v24_0_0 lsst_distrib.

Once you have set up the v24.0.0 version of the pipelines, you should also make sure you check out the v24 tag of rc2_subset.

After that, the tutorial at this link should work for you (note that it has v24_0_0 in the URL).

Let me/us know if you still have issues.

#5

Typing eups list -s lsst_distrib will show only the version that has been setup, making it easier to check which version that is. In Jeff’s command’s output, the word setup is more critical than the word current (which is the default version that will be setup if no other is specified).

#6

Thank you for your suggestions. Indeed, we will need to install v24.0.0 on one of our machine. There were problems in doing that in an Ubuntu OS, so we will try to install CentOS. This may take a while, I will come back to this as soon as possible.

Thank you again.

#7

Please let us know what problems you had on Ubuntu. While not an official platform, this should work, and we use Ubuntu ourselves in various places.

#8

On our machine the installation with the procedure described at:

https://pipelines.lsst.io/install/newinstall.html

Failed at different stages of the command:

eups distrib install -t v24_0_0 lsst_distrib

We tried with many different machines. At the end, what worked was the installation through the procedure at:

https://pipelines.lsst.io/install/lsstsw.html

This worked fine up to the tutorial 3. For the tutorial 4, we observed the problems described in this posts. We will try again with a machine operating CentOS. Please let me know if you have other questions.

#9

What error messages did you receive during eups distrib install?

#10

This is the latest error message, for package 31 afw:

[ 31/85 ] afw gb77f0a74f7+8a093cac5b …

***** error: from /home/valerio/SCRIPTS/LSST/lsst_track/stack/miniconda3-py38_4.9.2-5.0.0/EupsBuildDir/Linux64/afw-gb77f0a74f7+8a093cac5b/build.log:
428 | typename Ref<typename boost::mpl::at<PixelTVec, N>::type>::type apply_IMV(int x, int y) {
| ^~
include/lsst/afw/image/MaskedImage.h:428:65: error: template argument 1 is invalid
428 | typename Ref<typename boost::mpl::at<PixelTVec, N>::type>::type apply_IMV(int x, int y) {
| ^
In file included from src/detection/Psf.cc:7:
include/lsst/afw/detection/Psf.h: In member function ‘lsst::geom::Box2I lsst::afw::detection::Psf::computeKernelBBox() const’:
include/lsst/afw/detection/Psf.h:333:28: warning: ‘lsst::geom::Box2I lsst::afw::detection::Psf::computeBBox() const’ is deprecated: Default position argument overload is deprecated and will be removed in version 24.0. Please use overload with explicit position. [-Wdeprecated-declarations]
333 | return computeBBox();
| ^
include/lsst/afw/detection/Psf.h:297:23: note: declared here
297 | lsst::geom::Box2I computeBBox() const;
| ^~~~~~~~~~~
scons: *** [src/detection/Threshold.os] Error 1
scons: *** [src/detection/FootprintMerge.os] Error 1
scons: *** [src/detection/HeavyFootprint.os] Error 1
scons: *** [src/detection/Footprint.os] Error 1
scons: *** [src/detection/FootprintSet.os] Error 1
scons: building terminated because of errors.

  • exit -4
    eups distrib: Failed to build afw-gb77f0a74f7+8a093cac5b.eupspkg: Command:
    source “/home/valerio/miniconda3/envs/lsst-scipipe-5.0.0/eups/bin/setups.sh”; export EUPS_PATH="/home/valerio/SCRIPTS/LSST/lsst_track/stack/miniconda3-py38_4.9.2-5.0.0"; (/home/valerio/SCRIPTS/LSST/lsst_track/stack/miniconda3-py38_4.9.2-5.0.0/EupsBuildDir/Linux64/afw-gb77f0a74f7+8a093cac5b/build.sh) >> /home/valerio/SCRIPTS/LSST/lsst_track/stack/miniconda3-py38_4.9.2-5.0.0/EupsBuildDir/Linux64/afw-gb77f0a74f7+8a093cac5b/build.log 2>&1 4>/home/valerio/SCRIPTS/LSST/lsst_track/stack/miniconda3-py38_4.9.2-5.0.0/EupsBuildDir/Linux64/afw-gb77f0a74f7+8a093cac5b/build.msg
    exited with code 252

Sincerely,

#11

You seem to have the wrong version of the rubin-env environment. v24_0_0 should be built using lsst-scipipe-4.0.5, not 5.0.0. With that fixed, you can also try a binary install by doing
export EUPS_PKGROOT=https://eups.lsst.codes/stack/redhat/el7/conda-system/miniconda3-py38_4.9.2-4.0.5
before your eups distrib install command.

You could also try using lsstinstall as documented here: Install with lsstinstall and eups distrib — LSST Science Pipelines

#12

Hi,

We tried using version 24 for both the science pipelines and the rc_subset data, as suggested. Below there is the latest error message, what are we doing wrong? We will try again on a CentOS machine, but this will take some time…

Thank you,

lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/cmd/commands.py”, line 130, in run
qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/qgraph.py”, line 187, in qgraph
qgraph = f.makeGraph(pipelineObj, args)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 565, in makeGraph
butler, collections, run = _ButlerFactory.makeButlerAndCollections(args)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 352, in makeButlerAndCollections
butler, inputs, self = cls._makeReadParts(args)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 288, in _makeReadParts
self = cls(butler.registry, args, writeable=False)
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 218, in init
self.inputs = tuple(registry.queryCollections(args.input, flattenChains=True)) if args.input else ()
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registries/sql.py”, line 887, in queryCollections
for record in query.iter(
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registry/wildcards.py”, line 673, in iter
yield from self._search.iter(
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registry/wildcards.py”, line 507, in iter
manager.find(name),
File “/home/valerio/lsst_stack/conda/envs/lsst-scipipe-4.0.5/share/eups/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registry/collections/_base.py”, line 448, in find
raise MissingCollectionError(f"No collection with name ‘{name}’ found.")
lsst.daf.butler.registry._exceptions.MissingCollectionError: No collection with name ‘u/valerio/single_frame’ found.