Issue in running `pipetask` command

Hi everyone,

I am going through the LSST Science Pipelines tutorials and when I run the pipetask command (e.g. for singleFrame processing as presented here: Getting started tutorial part 2: calibrating single frames with Pipeline Tasks — LSST Science Pipelines ) I get the following error.

lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 119, in qgraph
    pipeline = script.build(**kwargs)
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/build.py", line 87, in build
    pipeline = f.makePipeline(args)
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 505, in makePipeline
    pipeline = Pipeline.from_uri(args.pipeline)
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipeline.py", line 330, in from_uri
    pipeline: Pipeline = cls.fromIR(pipelineIR.PipelineIR.from_uri(uri))
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipelineIR.py", line 897, in from_uri
    return cls(loaded_yaml)
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipelineIR.py", line 547, in __init__
    self._verify_labeled_subsets()
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipelineIR.py", line 606, in _verify_labeled_subsets
    raise ValueError(
ValueError: Labels {'updateVisitSummary'} were not found in the declared pipeline

I should mention that I do not get the above error when I comment lines 245 and 316 in rc2_subset/pipelines/DRP.py, which defines updateVisitSummary as a subset.

Could you please help me find out how to fix this issue and run the command without commenting updateVisitSummary in rc2_subset/pipelines/DRP.py

1 Like

What version of the pipeline software are you using? The version of rc2_subset you use depends on the associated version of the pipeline software. You should only use main branch if you are using a current weekly release (we have recently clarified that in the documentation but for the older releases they were not clear).

1 Like

To install the pipelines, I followed the instructions (1 to 4) from here: Install with newinstall.sh and eups distrib — LSST Science Pipelines

And to install git lfs properly, I followed the instructions from here: Problem constructing butler - #10 by ktl

git branch tells me I am in the main branch and that is the only branch available.

eups list lsst_distrib gives me this: g0b29ad24fb+8f7d15ecf4 current v24_0_0 v24_0_0_rc4 setup

Maybe git lfs fetch fetches the current weekly release content?

For the tutorial you need to use the 24.0.0 tag of rc2_subset if you are using v24 of science pipelines.

1 Like

git checkout 24.0.0 should do the right thing.

(git branch shows you local branches by default. There are more visible with git branch -r. But you want the tag, as Tim says, not a branch.)

Thank you both. Using 24.0.0 tag of rc2_subset resolved that problem.

Now, when I follow the instructions here: Getting started tutorial part 4: Using all the data to calibrate — LSST Science Pipelines , everything is good for FGCM and jointcal, but I get the following error when trying to apply the calibrations:

lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 129, in run
    pipeline = script.build(**kwargs)
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/build.py", line 87, in build
    pipeline = f.makePipeline(args)
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 505, in makePipeline
    pipeline = Pipeline.from_uri(args.pipeline)
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipeline.py", line 334, in from_uri
    pipeline = pipeline.subsetFromLabels(label_specifier)
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipeline.py", line 402, in subsetFromLabels
    return Pipeline.fromIR(self._pipelineIR.subset_from_labels(labelSet))
  File "/home/amir/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/pipe_base/g641d719c08+0e065976f8/python/lsst/pipe/base/pipelineIR.py", line 817, in subset_from_labels
    raise ValueError(
ValueError: Not all supplied labels (specified or named subsets) are in the pipeline definition, extra labels: {'source_calibration'}

Could you please help me to resolve this one too?

1 Like

@jeffcarlin is looking into the tutorial. Please confirm that you are using the 24.0.0 release of pipelines along with the 24.0.0 tag of rc2_subset?

Hi Amir -

It looks like the tutorial got updated with instructions for versions that come after v24 – apologies for the confusion.

I think that if you just skip the source_calibration step in the tutorial, it should work fine. (Don’t worry - the calibrations will still be applied, but it “just happens” under the hood without an explicit call.)

You could also follow the tutorial for pipelines version 23, which should work for v24 as well. In the meantime, I’ll get to work on identifying what went wrong in the v24 documentation.

1 Like

Sorry for my late response. I have moved to using the weekly version and got some distance from this. However, I decided to install v24_0_0 of the stack and find out if following the tutorials is straightforward and one can complete instructions there with the least problems.

Surprisingly, installing v24_0_0 fails when I follow the instructions in the tutorials. Here is the error I get when trying that:

  [  1/85 ]  alert_packet g5d8fbe4fe2 (already installed)               done. 
   [  2/85 ]  eigen g04a8d4365e (already installed)                      done. 
   [  3/85 ]  fgcm g941b12d670 (already installed)                       done. 
   [  4/85 ]  obs_decam_data gc3e517dea3 (already installed)             done. 
   [  5/85 ]  obs_lsst_data gbdb8a927be (already installed)              done. 
   [  6/85 ]  obs_subaru_data g2f68bc2906 (already installed)            done. 
   [  7/85 ]  proxmin g33b4157f25 (already installed)                    done. 
   [  8/85 ]  sconsUtils g7374e9d467 (already installed)                 done. 
   [  9/85 ]  sdm_schemas g8d82b3c5dd (already installed)                done. 
   [ 10/85 ]  spectractor g03aae25595 (already installed)                done. 
   [ 11/85 ]  astshim g38293774b4+ac198e9f13 (already installed)         done. 
   [ 12/85 ]  base gf9f5ea5b4d+ac198e9f13 (already installed)            done. 
   [ 13/85 ]  dustmaps_cachedata g41a3ec361e+ac198e9f13 (already installed)  done. 
   [ 14/85 ]  jointcal_cholmod ga68e3ac08d+ac198e9f13 (already installed)  done. 
   [ 15/85 ]  kht g14ffe67dc2+c057cea34b (already installed)             done. 
   [ 16/85 ]  psfex g57437a15a7+ac198e9f13 (already installed)           done. 
   [ 17/85 ]  scarlet gd32b658ba2+4083830bf8 (already installed)         done. 
   [ 18/85 ]  sphgeom ga1cf026fa3+ac198e9f13 (already installed)         done. 
   [ 19/85 ]  verify_metrics g40f75c44ca+ac198e9f13 (already installed)  done. 
   [ 20/85 ]  pex_exceptions g48ccf36440+89c08d0516 (already installed)  done. 
   [ 21/85 ]  scarlet_extensions g9d18589735+cc492336a9 (already installed)  done. 
   [ 22/85 ]  cpputils ga32aa97882+7403ac30ac (already installed)        done. 
   [ 23/85 ]  utils ga4dd45a4c7+82f6db4df0 (already installed)           done. 
   [ 24/85 ]  daf_base g5c4744a4d9+9e5e24d318 (already installed)        done. 
   [ 25/85 ]  geom g3b44f30a73+6ed7a0bf37 (already installed)            done. 
   [ 26/85 ]  log g9d27549199+9e5e24d318 (already installed)             done. 
   [ 27/85 ]  resources g73ff3781d8+9e5e24d318 (already installed)       done. 
  [ 28/85 ]  astro_metadata_translator g83a23aef33+846e1f9efd (already installed   [ 28/85 ]  astro_metadata_translator g83a23aef33+846e1f9efd (already installed)  done. 
   [ 29/85 ]  daf_butler gdde7253329+1ed234098f (already installed)      done. 
   [ 30/85 ]  pex_config gc75b51116a+846e1f9efd (already installed)      done. 
   [ 31/85 ]  afw gb77f0a74f7+8a093cac5b (already installed)             done. 
   [ 32/85 ]  daf_persistence g17e5ecfddb+2f99ec5bff (already installed)  done. 
   [ 33/85 ]  dax_apdb ga786bb30fb+e485989f06 (already installed)        done. 
   [ 34/85 ]  cbp g7177720bbd+f09366fa86 (already installed)             done. 
   [ 35/85 ]  display_ds9 gd01420fc67+f09366fa86 (already installed)     done. 
   [ 36/85 ]  display_firefly g1d67935e3f+f09366fa86 (already installed)  done. 
   [ 37/85 ]  display_matplotlib gbec6a3398f+f09366fa86 (already installed)  done. 
   [ 38/85 ]  pipe_base g641d719c08+0e065976f8 (already installed)       done. 
   [ 39/85 ]  shapelet gd877ba84e5+f09366fa86 (already installed)        done. 
   [ 40/85 ]  skymap gdb4cecd868+d155e19190 (already installed)          done. 
   [ 41/85 ]  coadd_utils gf3ee170dca+4a4456e283 (already installed)     done. 
   [ 42/85 ]  ctrl_mpexec g4d64b21cde+4a4456e283 (already installed)     done. 
   [ 43/85 ]  ctrl_pool g6c8d09e9e7+4a4456e283 (already installed)       done. 
   [ 44/85 ]  obs_base gbaa45dfa32+2718a75a08 (already installed)        done. 
   [ 45/85 ]  verify g77c5fecd56+422e7247c4 (already installed)          done. 
  [ 46/85 ]  ctrl_bps gca52d74647+a5413a7a82 ... 

***** error: from /Users/abazkiaei/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/EupsBuildDir/DarwinX86/ctrl_bps-gca52d74647+a5413a7a82/build.log:
ERROR python/lsst/ctrl/bps/restart.py::flake-8::FLAKE8 - _pytest.nodes.Collec...
ERROR python/lsst/ctrl/bps/submit.py::flake-8::FLAKE8 - _pytest.nodes.Collect...
ERROR python/lsst/ctrl/bps/version.py::flake-8::FLAKE8 - _pytest.nodes.Collec...
ERROR python/lsst/ctrl/bps/wms_service.py::flake-8::FLAKE8 - _pytest.nodes.Co...
ERROR python/lsst/ctrl/bps/cli/opt/arguments.py::flake-8::FLAKE8 - _pytest.no...
ERROR python/lsst/ctrl/bps/cli/bps.py::flake-8::FLAKE8 - _pytest.nodes.Collec...
ERROR python/lsst/ctrl/bps/transform.py::flake-8::FLAKE8 - _pytest.nodes.Coll...
ERROR python/lsst/ctrl/bps/cli/cmd/commands.py::flake-8::FLAKE8 - _pytest.nod...
ERROR python/lsst/ctrl/bps/cli/opt/option_groups.py::flake-8::FLAKE8 - _pytes...
ERROR python/lsst/ctrl/bps/cli/opt/options.py::flake-8::FLAKE8 - _pytest.node...
================= 20 passed, 20 warnings, 32 errors in 17.29s ==================
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /Users/abazkiaei/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/EupsBuildDir/DarwinX86/ctrl_bps-gca52d74647+a5413a7a82/ctrl_bps-gca52d74647+a5413a7a82/tests/.tests/pytest-ctrl_bps.xml.failed
The following tests failed:
/Users/abazkiaei/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/EupsBuildDir/DarwinX86/ctrl_bps-gca52d74647+a5413a7a82/ctrl_bps-gca52d74647+a5413a7a82/tests/.tests/pytest-ctrl_bps.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build ctrl_bps-gca52d74647+a5413a7a82.eupspkg: Command:
	source "/Users/abazkiaei/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.5/eups/bin/setups.sh"; export EUPS_PATH="/Users/abazkiaei/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5"; (/Users/abazkiaei/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/EupsBuildDir/DarwinX86/ctrl_bps-gca52d74647+a5413a7a82/build.sh) >> /Users/abazkiaei/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/EupsBuildDir/DarwinX86/ctrl_bps-gca52d74647+a5413a7a82/build.log 2>&1 4>/Users/abazkiaei/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/EupsBuildDir/DarwinX86/ctrl_bps-gca52d74647+a5413a7a82/build.msg 
exited with code 252

Hi @bazkiaei,
I’m not a pipeline expert so @timj or @jeffcarlin may clearer answer for you on this. But it seems like it’s crashing when trying to install “ctrl_bps” and it has some issue with flake8. I believe this is the github respository for that: GitHub - lsst/ctrl_bps: A PipelineTask execution framework for multi-node processing for the LSST Batch Production Service (BPS).

When searching that repo for any issues related to flake, I noticed this issue: DM-35971: Pytest requires flake<5 by mwittgen · Pull Request #117 · lsst/ctrl_bps · GitHub

It seems like the installation requires a version of flake8 lower than 5, so perhaps you could try reinstalling a lower version of flake8 and then giving it a try again?

Best,
Ryan

1 Like

It does look like the error we get from newer flake8 installations (you can look at the full failure by looking in the .xml.failed file). It might be worth to look in that failure file because it’s confusing that it got all the way to ctrl_bps before it became a problem. A newer flake8 should have caused failures almost immediately.

It’s entirely possible that the rubin-env created for the v24.0 release is no longer compatible but I can’t explain why it took so long to manifest.

1 Like

Thank you both.
Installing flake8 with a version lower than 5 did not fix the problem and I got the same error. Here I attach the .xml.failed files. (the file starting with 1 contains the error before lowering the flake8 version and the file starting with 2 contains the error I got after lowering the flake8 version.)

1_pytest-ctrl_bps.xml.failed (941.6 KB)

2_pytest-ctrl_bps.xml.failed (931.6 KB)

And may I ask, in case the rubin-env for v24.0 release is no longer compatible, which rubin-env and stack release do you recommend for following the tutorials?

OK, I may be slightly out of my depth here, but I’ll share my thoughts on this and perhaps @timj or @jeffcarlin can jump in later.

It seems like at least the first error that pops up is related to the “read_gpickle” function from the “networkx” module. It looks like that function as well as “write_gpickle” were deprecated networkx functions, and perhaps that’s why it can’t find those. Interestingly, it seems this should have been removed in “python/lsst/ctrl/bps/generic_workflow.py” based on this commit: Remove deprecated NetworkX functions · lsst/ctrl_bps@5e1bb28 · GitHub

Indeed the generic_workflow.py script on the main version of ctrl_bps (ctrl_bps/generic_workflow.py at main · lsst/ctrl_bps · GitHub) seems to have those _gpickle function removed from the networkx imports. Maybe trying to install v24 reads in an older version that is indeed no longer compatible.

I don’t currently have any advice on how to proceed, but hopefully those more informed can provide some more guidance.

That explains it. We have fixed the networkx problem in the soon-to-be-released v24.1. For now you should downgrade the networkx in your conda environment.

These errors are the networkx problem reported by @ryanlau . Downgrading networkx will fix it. 24.1 should be out soon.