FGCM too many indices error

Support
#1

Hi all,

I am currently trying to run fgcm on my data and am running into the following error at the fgcmBuildFromIsolatedStarsTask step:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 473, in _executeQuantaInProcess
    self.quantumExecutor.execute(qnode.taskDef, qnode.quantum)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 161, in execute
    result = self._execute(taskDef, quantum)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 258, in _execute
    self.runQuantum(task, quantum, taskDef, limited_butler)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 460, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/fgcmcal/g98cecbdb62+ed2cb6d659/python/lsst/fgcmcal/fgcmBuildFromIsolatedStars.py", line 246, in runQuantum
    struct = self.run(
             ^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/fgcmcal/g98cecbdb62+ed2cb6d659/python/lsst/fgcmcal/fgcmBuildFromIsolatedStars.py", line 331, in run
    fgcm_ref = self._associate_reference_stars(lookup_table_handle, visit_cat, fgcm_obj)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/fgcmcal/g98cecbdb62+ed2cb6d659/python/lsst/fgcmcal/fgcmBuildFromIsolatedStars.py", line 749, in _associate_reference_stars
    ref_cat = self.fgcmLoadReferenceCatalog.getFgcmReferenceStarsSkyCircle(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/fgcmcal/g98cecbdb62+ed2cb6d659/python/lsst/fgcmcal/fgcmLoadReferenceCatalog.py", line
232, in getFgcmReferenceStarsSkyCircle
    fgcmRefCat['refMag'][:, :] = 99.0
    ~~~~~~~~~~~~~~~~~~~~^^^^^^
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 0 quanta successfully, 1 failed and 2 remain out of total 3 quanta.
lsst.ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(lsst.fgcmcal.fgcmFitCycle.FgcmFitCycleTask, label=fgcmFitCycle) dataId={instrument: 'HSC'}>, skipping this task.
lsst.ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(lsst.fgcmcal.fgcmOutputProducts.FgcmOutputProductsTask, label=fgcmOutputProducts) dataId={instrument: 'HSC'}>, skipping this task.
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 290, in run_qbb
    script.run_qbb(repo, qgraph, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_
mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/script/run_qbb.py", line 115, in run_qbb
    f.runGraphQBB(task_factory, args)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 982, in runGraphQBB
    executor.execute(qgraph)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 405, in execute
    self._executeQuantaInProcess(graph, self.report)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 529, in _executeQuantaInProcess
    raise MPGraphExecutorError("One or more tasks failed during execution.")
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution.

Does anyone know how to fix this error, or what I might be doing wrong? For context, I am running v26 of the pipeline.

#2

Hi Claire,

I found a potentially similar error message discussed in this other forum topic:

Which was solved by noting that (at the time) there were differences between running FGCM on a single-band data set versus a multi-band data set:

Perhaps you could have a look at that topic and then let us know how much the info there helps with the error you’re currently encountering? Thanks very much.

#3

Hi Aaron,

Thanks for the tip. I was able to un jointcal on the data, as per the the other forum post, and it worked. Thank you for the suggestion! Just so that I understand, does fgcm always need to be run on a multi-band dataset?

#4

Hi Claire – Yes, FGCM needs to always be run on a multi-band data set. Thanks…

1 Like