lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 119, in qgraph
pipeline = script.build(**kwargs)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/build.py", line 87, in build
pipeline = f.makePipeline(args)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 505, in makePipeline
pipeline = Pipeline.from_uri(args.pipeline)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipeline.py", line 330, in from_uri
pipeline: Pipeline = cls.fromIR(pipelineIR.PipelineIR.from_uri(uri))
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipelineIR.py", line 897, in from_uri
return cls(loaded_yaml)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipelineIR.py", line 547, in __init__
self._verify_labeled_subsets()
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipelineIR.py", line 606, in _verify_labeled_subsets
raise ValueError(
ValueError: Labels {'updateVisitSummary'} were not found in the declared pipeline
I should mention that I do not get the above error when I comment lines 245 and 316 in rc2_subset/pipelines/DRP.py, which defines updateVisitSummary as a subset.
Could you please help me find out how to fix this issue and run the command without commenting updateVisitSummary in rc2_subset/pipelines/DRP.py
What version of the pipeline software are you using? The version of rc2_subset you use depends on the associated version of the pipeline software. You should only use main branch if you are using a current weekly release (we have recently clarified that in the documentation but for the older releases they were not clear).
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 129, in run
pipeline = script.build(**kwargs)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/build.py", line 87, in build
pipeline = f.makePipeline(args)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 505, in makePipeline
pipeline = Pipeline.from_uri(args.pipeline)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipeline.py", line 334, in from_uri
pipeline = pipeline.subsetFromLabels(label_specifier)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipeline.py", line 402, in subsetFromLabels
return Pipeline.fromIR(self._pipelineIR.subset_from_labels(labelSet))
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipelineIR.py", line 817, in subset_from_labels
raise ValueError(
ValueError: Not all supplied labels (specified or named subsets) are in the pipeline definition, extra labels: {'source_calibration'}
It looks like the tutorial got updated with instructions for versions that come after v24 – apologies for the confusion.
I think that if you just skip the source_calibration step in the tutorial, it should work fine. (Don’t worry - the calibrations will still be applied, but it “just happens” under the hood without an explicit call.)
You could also follow the tutorial for pipelines version 23, which should work for v24 as well. In the meantime, I’ll get to work on identifying what went wrong in the v24 documentation.
Sorry for my late response. I have moved to using the weekly version and got some distance from this. However, I decided to install v24_0_0 of the stack and find out if following the tutorials is straightforward and one can complete instructions there with the least problems.
Surprisingly, installing v24_0_0 fails when I follow the instructions in the tutorials. Here is the error I get when trying that:
It seems like the installation requires a version of flake8 lower than 5, so perhaps you could try reinstalling a lower version of flake8 and then giving it a try again?
It does look like the error we get from newer flake8 installations (you can look at the full failure by looking in the .xml.failed file). It might be worth to look in that failure file because it’s confusing that it got all the way to ctrl_bps before it became a problem. A newer flake8 should have caused failures almost immediately.
It’s entirely possible that the rubin-env created for the v24.0 release is no longer compatible but I can’t explain why it took so long to manifest.
Thank you both.
Installing flake8 with a version lower than 5 did not fix the problem and I got the same error. Here I attach the .xml.failed files. (the file starting with 1 contains the error before lowering the flake8 version and the file starting with 2 contains the error I got after lowering the flake8 version.)
And may I ask, in case the rubin-env for v24.0 release is no longer compatible, which rubin-env and stack release do you recommend for following the tutorials?
OK, I may be slightly out of my depth here, but I’ll share my thoughts on this and perhaps @timj or @jeffcarlin can jump in later.
It seems like at least the first error that pops up is related to the “read_gpickle” function from the “networkx” module. It looks like that function as well as “write_gpickle” were deprecated networkx functions, and perhaps that’s why it can’t find those. Interestingly, it seems this should have been removed in “python/lsst/ctrl/bps/generic_workflow.py” based on this commit: Remove deprecated NetworkX functions · lsst/ctrl_bps@5e1bb28 · GitHub
Indeed the generic_workflow.py script on the main version of ctrl_bps (ctrl_bps/generic_workflow.py at main · lsst/ctrl_bps · GitHub) seems to have those _gpickle function removed from the networkx imports. Maybe trying to install v24 reads in an older version that is indeed no longer compatible.
I don’t currently have any advice on how to proceed, but hopefully those more informed can provide some more guidance.
That explains it. We have fixed the networkx problem in the soon-to-be-released v24.1. For now you should downgrade the networkx in your conda environment.
While installing the stack went well I got some problems running stack following the tutorial.
The #single_frame, #fgcm, and #jointcal steps work well when following the tutorial. Following the advice from @jeffcarlin , I ignored the #source_calibration step to skip the error that running that step raises (you can see the error here: Issue in running `pipetask` command - #6 by bazkiaei).
The next step in the tutorial is running #makeWarp. When I run that I get the following error:
(lsst-scipipe-4.0.5) amir@SCI-10071 ~/lsst_stack $ pipetask run -b /home/amir/lsst_stack/rc2_subset/SMALL_HSC/butler.yaml -d "tract = 9813 AND skymap = 'hsc_rings_v1' AND patch in (38, 39, 40, 41)" -p /home/amir/lsst_stack/rc2_subset/pipelines/DRP.yaml#makeWarp -i u/amir/jointcal,u/amir/fgcm -o u/amir/warps --register-dataset-types
lsst.pipe.base.graphBuilder WARNING: Dataset type finalized_psf_ap_corr_catalog is not registered.
py.warnings WARNING: /home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/qgraph.py:187: UserWarning: QuantumGraph is empty
qgraph = f.makeGraph(pipelineObj, args)
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 130, in run
qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 190, in qgraph
raise RuntimeError("QuantumGraph is empty.")
RuntimeError: QuantumGraph is empty.
Hi, I have been following this trend because I was having the same issues installing eups distrib install -t v24_0_0 lsst_distrib. I downgraded networkx as following: mamba install networkx==2.5 (I hope I did that correctly). I tried again the installation, passed the 46/85 step (the one showing the networkx problem) but it failed in the 47/85. I attached the error message for the fail test. I hope you can help me. 1_pytest-ctrl_bps.xml.failed (941.6 KB)
It seems like something has gone wrong because your error is still the same error. If you look in the .failed file you will see:
ImportError while importing test module '/Users/abazkiaei/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/EupsBuildDir/DarwinX86/ctrl_bps-gca52d74647+a5413a7a82/ctrl_bps-gca52d74647+a5413a7a82/tests/test_bps_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../../../../conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.5/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_bps_utils.py:26: in <module>
from lsst.ctrl.bps.bps_utils import chdir
python/lsst/ctrl/bps/__init__.py:29: in <module>
from .generic_workflow import *
python/lsst/ctrl/bps/generic_workflow.py:35: in <module>
from networkx import DiGraph, read_gpickle, topological_sort, write_gpickle
E ImportError: cannot import name 'read_gpickle' from 'networkx'
What does mamba list | grep networkx report for you?