Issues with ingestCalib.py using DECAM images

Thanks a lot for your response Meredith! I was wondering if there are any command tags for pipetask run that could “continue” a run that was stopped due to computer-related issues. I’ve seen the pipetask run tags, and It’s not clear for me which one could do that or if any of them can. Many times a pipetask run stops (due to computer issues) and I find it a little inefficient to erase the crosstalk-sources folder and start again.
I’ve tried adding --skip-existing --extend-run but it looks like it’s not intended for that purpose :confused:
Thanks!

I think --skip-existing --extend-run should do what you describe. What is happening instead?

One idea: try not specifying input collections the second time you call pipetask run; once input collections are specified, they can’t be changed for subsequent reruns to the same output collection.

1 Like

when I added --skip-existing --extend-run to the command line, I got:

  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 102, in run
    qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 145, in qgraph
    qgraph = f.makeGraph(pipelineObj, args)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 550, in makeGraph
    qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query, metadata=metadata)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/graphBuilder.py", line 904, in makeGraph
    scaffolding.resolveDatasetRefs(self.registry, collections, run, commonDataIds,
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/graphBuilder.py", line 718, in resolveDatasetRefs
    raise OutputExistsError(
lsst.pipe.base.graphBuilder.OutputExistsError: Quantum {instrument: 'DECam', detector: 56, exposure: 411752, ...} of task with label 'overscan' has some outputs that exist ([DatasetRef(DatasetType('overscanRaw', {band, instrument, detector, physical_filter, exposure}, Exposure), {instrument: 'DECam', detector: 56, exposure: 411752, ...}, id=b2d0c20c-fb4c-4e57-b1bb-cd360763cc01, run='DECam/raw/crosstalk-sources/20210811T065927Z')]) and others that don't ([DatasetRef(DatasetType('overscan_metadata', {band, instrument, detector, physical_filter, exposure}, PropertySet), {instrument: 'DECam', detector: 56, exposure: 411752, ...})]), with no metadata output, and clobbering outputs was not enabled.

I will try not to specify the input collection next time! and will give you updates on that once I try it out. For now, I was able to do the pipetask run and was wondering about your nightly build:

In my case I ran:

butler query-dimension-records /mnt/flock/jahumada/data_hits exposure --where "instrument='DECam' AND exposure.observation_type='bias' AND exposure.exposure_time=0.0 AND exposure.day_obs=20150313"

What could mean that this command line yields no results? Do you know of a command tag that could show me the available bias exposures without building anything yet? Something like --show --where "instrument='DECam' AND exposure.observation_type='bias' AND exposure.exposure_time=0.0"

Thanks for your time :slight_smile:

and clobbering outputs was not enabled.

The error message tells you how to fix it. Looking at the help you will find --clobber-outputs.

That is the right query fragment to find bias observations. You shouldn’t need to also have to specify the exposure time.

Do you know that you ingested some raw bias exposures? Maybe the biases were from different days?

Hi,

I will consider that next time as well :slight_smile:

I ingested the calib images using butler ingest-raws . There were no problems and it said that it ingested successfully. The output of butler ingest-raws of the calibration and raw images was:

Name                     Type
------------------------------------ -----------
DECam/calib                          CALIBRATION
DECam/calib/unbounded                RUN
DECam/calib/curated/19700101T000000Z RUN
DECam/calib/curated/20130115T013000Z RUN
DECam/calib/curated/20130916T092600Z RUN
DECam/calib/curated/20140117T012900Z RUN
DECam/calib/curated/20141020T103000Z RUN
DECam/calib/curated/20150105T011500Z RUN
DECam/calib/curated/20131130T000000Z RUN

What’s a little confusing is the dates of the DECam/calib/curated/ , because all the images I’m working with are between 2014 and 2015. Could this be why I have no results with butler query-dimension-records?

I also tried the following:

(lsst-scipipe-0.6.0) [jahumada@leftraru3 jahumada]$ butler query-dimension-records /mnt/flock/jahumada/data_hits exposure  --where "instrument='DECam' AND exposure.observation_type='bias' AND exposure.day_obs>20150313"
No results. Try --help for more information.

(lsst-scipipe-0.6.0) [jahumada@leftraru3 jahumada]$ butler query-dimension-records /mnt/flock/jahumada/data_hits exposure  --where "instrument='DECam' AND exposure.observation_type='bias' AND exposure.day_obs<20150313"
No results. Try --help for more information.

It seems that the butler doesn’t recognize any bias type image.

PS: I am also using the weekly LSST Science Pipelines version nº27, I’ve had it before the big release of version 22 was available, I am not sure if having this version could produce these types of errors I am having.

Thanks!

Yes. These are the curated calibrations (such as defect masks) that are small and are added to a butler repository with the butler write-curated-calibrations command. These will not include bias or dark images.

The raws you ingest will end up in collect DECam/raw/all – what raw files did you ask to ingest with butler ingest-raws? If you only ingested data from one of our general testdata repositories those will likely all be science frames. You can see what raw data you have by not constraining that butler query-dimension-records command by observation type. (or by using butler query-datasets with the raw dataset type).

1 Like

Ok :slight_smile: I found the problem,
Putting zero instead of bias to the observation_type, I got results :slight_smile:

(lsst-scipipe-0.6.0) jahumada@leftraru1:/mnt/flock/jahumada$ butler query-dimension-records /mnt/flock/jahumada/data_hits exposure  --where "instrument='DECam' AND exposure.observation_type='zero' AND exposure.day_obs=20150313"
instrument   id     physical_filter          obs_id       exposure_time dark_time observation_type observation_reason day_obs  seq_num group_name group_id  target_name   science_program    tracking_ra         tracking_dec    sky_angle zenith_angle                       timespan [2]                      

 DECam 421350 solid plate 0.0 0.0 ct4m20150313t193722           0.0 0.0139599             zero            unknown 20150313  421350     421350   421350  preflats-BIAS      2015A-0608 34.380787207985456  20.708306035300883      90.0         None 2015-03-13 19:37:57.656041 .. 2015-03-13 19:38:48.000000
 DECam 421351 solid plate 0.0 0.0 ct4m20150313t193745           0.0 0.0122199             zero            unknown 20150313  421351     421351   421351  preflats-BIAS      2015A-0608  34.47681637530135  20.708389100893545      90.0         None 2015-03-13 19:38:20.610996 .. 2015-03-13 19:39:07.000000
 DECam 421352 solid plate 0.0 0.0 ct4m20150313t193808           0.0 0.0137498             zero            unknown 20150313  421352     421352   421352  preflats-BIAS      2015A-0608  34.57286220928707  20.708499944262773      90.0         None 2015-03-13 19:38:43.844598 .. 2015-03-13 19:39:31.000000
 DECam 421353 solid plate 0.0 0.0 ct4m20150313t193832           0.0   0.01334             zero            unknown 20150313  421353     421353   421353  preflats-BIAS      2015A-0608 34.668908043285676  20.708556065406825      90.0         None 2015-03-13 19:39:07.340203 .. 2015-03-13 19:39:52.000000
 DECam 421354 solid plate 0.0 0.0 ct4m20150313t193855           0.0 0.0148501             zero            unknown 20150313  421354     421354   421354  preflats-BIAS      2015A-0608  34.76912054398033  20.708667186982925      90.0         None 2015-03-13 19:39:30.318561 .. 2015-03-13 19:40:16.000000
 DECam 421355 solid plate 0.0 0.0 ct4m20150313t193918           0.0    0.0144             zero            unknown 20150313  421355     421355   421355  preflats-BIAS      2015A-0608  34.86514971132854  20.708694974785754      90.0         None 2015-03-13 19:39:53.326389 .. 2015-03-13 19:40:48.000000
 DECam 421356 solid plate 0.0 0.0 ct4m20150313t193941           0.0   0.03789             zero            unknown 20150313  421356     421356   421356  preflats-BIAS      2015A-0608 34.961195545336615   20.70883415147616      90.0         None 2015-03-13 19:40:16.304359 .. 2015-03-13 19:41:02.000000
 DECam 421357 solid plate 0.0 0.0 ct4m20150313t194004           0.0 0.0140901             zero            unknown 20150313  421357     421357   421357  preflats-BIAS      2015A-0608  35.05724137935772   20.70891721705238      90.0         None 2015-03-13 19:40:39.543707 .. 2015-03-13 19:41:27.000000
 DECam 421358 solid plate 0.0 0.0 ct4m20150313t194027           0.0 0.0134001             zero            unknown 20150313  421358     421358   421358  preflats-BIAS      2015A-0608  35.15328721338843   20.70897306040326      90.0         None 2015-03-13 19:41:02.570380 .. 2015-03-13 19:41:48.000000
 DECam 421359 solid plate 0.0 0.0 ct4m20150313t194050           0.0 0.0127599             zero            unknown 20150313  421359     421359   421359  preflats-BIAS      2015A-0608  35.24931638075213   20.70908418152705      90.0         None 2015-03-13 19:41:25.520977 .. 2015-03-13 19:42:11.000000
 DECam 421360 solid plate 0.0 0.0 ct4m20150313t194113           0.0 0.0136201             zero            unknown 20150313  421360     421360   421360  preflats-BIAS      2015A-0608  35.34540804812597   20.70916724709863      90.0         None 2015-03-13 19:41:48.609948 .. 2015-03-13 19:42:34.000000
 DECam 421448 solid plate 0.0 0.0 ct4m20150313t204838           0.0   0.01372             zero            unknown 20150313  421448     421448   421448 postflats-BIAS      2015A-0608  52.23791243511519   20.72661282207203      90.0         None 2015-03-13 20:49:13.651863 .. 2015-03-13 20:49:59.000000
 DECam 421449 solid plate 0.0 0.0 ct4m20150313t204901           0.0   0.01579             zero            unknown 20150313  421449     421449   421449 postflats-BIAS      2015A-0608  52.33394160356247   20.72672477553536      90.0         None 2015-03-13 20:49:36.530961 .. 2015-03-13 20:50:22.000000
 DECam 421450 solid plate 0.0 0.0 ct4m20150313t204924           0.0   0.01406             zero            unknown 20150313  421450     421450   421450 postflats-BIAS      2015A-0608  52.43003327201389  20.726862840107717      90.0         None 2015-03-13 20:49:59.556428 .. 2015-03-13 20:50:54.000000
 DECam 421451 solid plate 0.0 0.0 ct4m20150313t204947           0.0   0.04866             zero            unknown 20150313  421451     421451   421451 postflats-BIAS      2015A-0608   52.5260749404759  20.726946738000716      90.0         None 2015-03-13 20:50:22.697510 .. 2015-03-13 20:51:08.000000
 DECam 421452 solid plate 0.0 0.0 ct4m20150313t205010           0.0 0.0144701             zero            unknown 20150313  421452     421452   421452 postflats-BIAS      2015A-0608 52.622120775604465  20.727085913663895      90.0         None 2015-03-13 20:50:45.748267 .. 2015-03-13 20:51:31.000000
 DECam 421453 solid plate 0.0 0.0 ct4m20150313t205033           0.0 0.0120902             zero            unknown 20150313  421453     421453   421453 postflats-BIAS      2015A-0608    52.718195777409   20.72719675598844      90.0         None 2015-03-13 20:51:08.777096 .. 2015-03-13 20:51:54.000000
 DECam 421454 solid plate 0.0 0.0 ct4m20150313t205056           0.0 0.0141101             zero            unknown 20150313  421454     421454   421454 postflats-BIAS      2015A-0608  52.81428744588622  20.727307876084247      90.0         None 2015-03-13 20:51:31.865202 .. 2015-03-13 20:52:17.000000
 DECam 421455 solid plate 0.0 0.0 ct4m20150313t205120           0.0   0.01296             zero            unknown 20150313  421455     421455   421455 postflats-BIAS      2015A-0608  52.91031661436798   20.72741899616613      90.0         None 2015-03-13 20:51:55.012157 .. 2015-03-13 20:52:40.000000
 DECam 421456 solid plate 0.0 0.0 ct4m20150313t205143           0.0 0.0128601             zero            unknown 20150313  421456     421456   421456 postflats-BIAS      2015A-0608  53.00636244952244   20.72752983846369      90.0         None 2015-03-13 20:52:18.092811 .. 2015-03-13 20:53:03.000000
 DECam 421457 solid plate 0.0 0.0 ct4m20150313t205206           0.0   0.01439             zero            unknown 20150313  421457     421457   421457 postflats-BIAS      2015A-0608 53.102449951347076  20.727669014090335      90.0         None 2015-03-13 20:52:41.109736 .. 2015-03-13 20:53:27.000000
 DECam 421458 solid plate 0.0 0.0 ct4m20150313t205229           0.0 0.0143199             zero            unknown 20150313  421458     421458   421458 postflats-BIAS      2015A-0608  53.19849578651594  20.727751800816037      90.0         None 2015-03-13 20:53:04.075676 .. 2015-03-13 20:53:50.000000
 DECam 421459 solid plate 0.0 0.0 ct4m20150313t234600           0.0 0.0106499             zero            unknown 20150313  421459     421459   421459                     2015A-0608  91.52315333295033  -29.75860440698451      90.0         None 2015-03-13 23:46:35.384677 .. 2015-03-13 23:47:21.000000
 DECam 421460 solid plate 0.0 0.0 ct4m20150313t234623           0.0   0.01213             zero            unknown 20150313  421460     421460   421460                     2015A-0608  91.52315333295188 -29.758632462540064      90.0         None 2015-03-13 23:46:58.273867 .. 2015-03-13 23:47:44.000000

Thanks for the tips!

Hi all! :slight_smile:
I have ran the command line to build a master bias for the observation night of 2014-02-28:

pipetask run -b data_hits/ -p $CP_PIPE_DIR/pipelines/cpBias.yaml -i DECam/raw/all,DECam/calib -o hits_master_calib/on20140228_bias -d "instrument='DECam' AND exposure.observation_type='zero' AND exposure.day_obs=20140228" -j 3 --register-dataset-types

Supposedly, It ran successfully. Here is one of the last output lines I obtained:

ctrl.mpexec.mpGraphExecutor INFO: Executed 1485 quanta successfully, 0 failed and 3 remain out of total 1488 quanta.

isr WARN: No rough magnitude zero point defined for filter solid plate 0.0 0.0.

isr WARN: Non-positive exposure time; skipping rough zero point.

ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'isr' on quantum {instrument: 'DECam', detector: 27, exposure: 288902, ...} took 5.818 seconds

ctrl.mpexec.mpGraphExecutor INFO: Executed 1486 quanta successfully, 0 failed and 2 remain out of total 1488 quanta.

ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'isr' on quantum {instrument: 'DECam', detector: 27, exposure: 288832, ...} took 5.808 seconds

ctrl.mpexec.mpGraphExecutor INFO: Executed 1487 quanta successfully, 0 failed and 1 remain out of total 1488 quanta.

cpBiasCombine INFO: Scaling input 0 by 1.0

cpBiasCombine INFO: Scaling input 1 by 1.0

cpBiasCombine INFO: Scaling input 2 by 1.0

cpBiasCombine INFO: Scaling input 3 by 1.0

cpBiasCombine INFO: Scaling input 4 by 1.0

cpBiasCombine INFO: Scaling input 5 by 1.0

cpBiasCombine INFO: Scaling input 6 by 1.0

cpBiasCombine INFO: Scaling input 7 by 1.0

cpBiasCombine INFO: Scaling input 8 by 1.0

cpBiasCombine INFO: Scaling input 9 by 1.0

cpBiasCombine INFO: Scaling input 10 by 1.0

cpBiasCombine INFO: Scaling input 11 by 1.0

cpBiasCombine INFO: Scaling input 12 by 1.0

cpBiasCombine INFO: Scaling input 13 by 1.0

cpBiasCombine INFO: Scaling input 14 by 1.0

cpBiasCombine INFO: Scaling input 15 by 1.0

cpBiasCombine INFO: Scaling input 16 by 1.0

cpBiasCombine INFO: Scaling input 17 by 1.0

cpBiasCombine INFO: Scaling input 18 by 1.0

cpBiasCombine INFO: Scaling input 19 by 1.0

cpBiasCombine INFO: Scaling input 20 by 1.0

cpBiasCombine INFO: Scaling input 21 by 1.0

cpBiasCombine INFO: Scaling input 22 by 1.0

cpBiasCombine WARN: Found 8025961 NAN pixels

cpBiasCombine WARN: Exception making an obs group for headers. Continuing.

ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'cpBiasCombine' on quantum {instrument: 'DECam', detector: 27} took 30.350 seconds

ctrl.mpexec.mpGraphExecutor INFO: Executed 1488 quanta successfully, 0 failed and 0 remain out of total 1488 quanta.

But the thing is that I don’t have an output image on the hits_master_calib/on20140228_bias folder, so I am a little confused as to what the outcome should be. Maybe I should add a configuration tag -c to obtain the master bias? If so, I’m not sure how It must be configured :open_mouth:
Thanks a lot for your time again!

This is not a “folder”, it’s a Butler “collection”. In particular, with this command line, your output will go into a “RUN” collection that is named with a timestamp that is then chained into the output collection that you specified.

You can use butler command line tools to look at your collections and the datasets within them. You can also specify your output collection as an input collection for other pipetask run commands. Finally, if you want the actual file containing the master bias output, the safest mechanism is to use butler retrieve-artifacts, although you can also look in a folder/directory named after the time-stamped collection under your Butler datastore root.

1 Like

Thanks for your indications! I was able to found where the master biases are being generated, as well as for the flats.
Now that I have the master bias and the flats of a night of observation, I would like to do the calibrations of the science images. Following somewhat the indications of the tutorial of Getting Started with the HSC camera, I should do a processccd task for my DECam images.
One of the inputs of the processccd task is a reference catalog, that I don’t have. I should ingest a Pan-STARRS catalog for it to work, right?

I’ve been trying to run the following:

pipetask run -b data_hits/ -p $AP_PIPE_DIR/pipelines/DarkEnergyCamera/ProcessCcd.yaml -i DECam/raw/all,hits_master_calib/on20140228_bias,hits_master_calib/on20140228_flats --instrument lsst.obs.decam.DarkEnergyCamera --output-run processCcdOutputs/pon20140228

The night that I want to process is 2014-02-28, and the directories hits_master_calib/on20140228_bias,hits_master_calib/on20140228_flats correspond to the folders where the master bias and “master” flat is located (these directories are in the butler query-collections).

Running this command I get an error that could also be due to the lack of a reference catalog.
The error is this:

numexpr.utils INFO: Note: NumExpr detected 20 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
numexpr.utils INFO: NumExpr defaulting to 8 threads.
Traceback (most recent call last):
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/bin/pipetask", line 29, in <module>
    sys.exit(main())
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cli/pipetask.py", line 43, in main
    return cli()
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 102, in run
    qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 145, in qgraph
    qgraph = f.makeGraph(pipelineObj, args)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 550, in makeGraph
    qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query, metadata=metadata)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/graphBuilder.py", line 892, in makeGraph
    scaffolding = _PipelineScaffolding(pipeline, registry=self.registry)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/graphBuilder.py", line 411, in __init__
    datasetTypes = PipelineDatasetTypes.fromPipeline(pipeline, registry=registry)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/pipeline.py", line 910, in fromPipeline
    for taskDef in pipeline:
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/pipeline.py", line 576, in toExpandedPipeline
    overrides.applyTo(config)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/configOverrides.py", line 296, in applyTo
    instrument.applyConfigOverrides(name, config)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/obs_base/21.0.0-60-g2baedfa+3512c75b65/python/lsst/obs/base/_instrument.py", line 346, in applyConfigOverrides
    config.load(path)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_config/21.0.0-1-ga51b5d4+9915f4c1f0/python/lsst/pex/config/config.py", line 1036, in load
    self.loadFromStream(stream=code, root=root, filename=filename)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_config/21.0.0-1-ga51b5d4+9915f4c1f0/python/lsst/pex/config/config.py", line 1075, in loadFromStream
    exec(stream, globals, local)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/obs_decam/21.0.0-20-g1e553c2+458274d3ec/config/calibrate.py", line 30, in <module>
    config.measurement.load(os.path.join(obsConfigDir, "hsm.py"))
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_config/21.0.0-1-ga51b5d4+9915f4c1f0/python/lsst/pex/config/config.py", line 1036, in load
    self.loadFromStream(stream=code, root=root, filename=filename)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_config/21.0.0-1-ga51b5d4+9915f4c1f0/python/lsst/pex/config/config.py", line 1075, in loadFromStream
    exec(stream, globals, local)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/obs_decam/21.0.0-20-g1e553c2+458274d3ec/config/hsm.py", line 8, in <module>
    config.load(os.path.join(getPackageDir("meas_extensions_shapeHSM"), "config", "enable.py"))
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_config/21.0.0-1-ga51b5d4+9915f4c1f0/python/lsst/pex/config/config.py", line 1036, in load
    self.loadFromStream(stream=code, root=root, filename=filename)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_config/21.0.0-1-ga51b5d4+9915f4c1f0/python/lsst/pex/config/config.py", line 1075, in loadFromStream
    exec(stream, globals, local)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/config/enable.py", line 7, in <module>
    import lsst.meas.extensions.shapeHSM
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/python/lsst/meas/extensions/shapeHSM/__init__.py", line 27, in <module>
    from .hsmMomentsControl import *
ImportError: libgalsim.so.2.2: cannot open shared object file: No such file or directory

Thanks a lot for your time! :slight_smile:

That’s a strange error that indicates problems with your conda environment, not something to do with anything astronomical. Can you please post the results of these commands?

$ conda list galsim
$ conda list rubin-env
$ ls -l $CONDA_PREFIX/lib/galsim*
$ ldd $EUPS_PATH/Linux64/meas_extensions_shapeHSM/*/lib/libmeas_extensions_shapeHSM.so

Ok!, here is what I got:

(lsst-scipipe-0.6.0) [jahumada@leftraru3 jahumada]$ conda list galsim
# packages in environment at /home/jahumada/.conda/envs/lsst-scipipe-0.6.0:
#
# Name                    Version                   Build  Channel
galsim                    2.3.0            py38h260733e_0    conda-forge
(lsst-scipipe-0.6.0) [jahumada@leftraru3 jahumada]$ conda list rubin-env
# packages in environment at /home/jahumada/.conda/envs/lsst-scipipe-0.6.0:
#
# Name                    Version                   Build  Channel
rubin-env                 0.6.0                ha770c72_1    conda-forge
rubin-env-nosysroot       0.6.0                ha770c72_1    conda-forge
(lsst-scipipe-0.6.0) [jahumada@leftraru3 jahumada]$ ls -l $CONDA_PREFIX/lib/galsim*
ls: cannot access /home/jahumada/.conda/envs/lsstscipipe-0.6.0/lib/galsim*: No such file or directory
(lsst-scipipe-0.6.0) [jahumada@leftraru3 jahumada]$ ldd $EUPS_PATH/Linux64/meas_extensions_shapeHSM/*/lib/libmeas_extensions_shapeHSM.so

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-14-g3bd782b+16b33694c2/lib/libmeas_base.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-14-g3bd782b+16b33694c2/lib/libmeas_base.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-14-g3bd782b+16b33694c2/lib/libmeas_base.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-14-g3bd782b+16b33694c2/lib/libmeas_base.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-46-g27369044f+532d44eec1/lib/libafw.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-46-g27369044f+532d44eec1/lib/libafw.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-46-g27369044f+532d44eec1/lib/libafw.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-46-g27369044f+532d44eec1/lib/libafw.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-46-g27369044f+532d44eec1/lib/libafw.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/geom/21.0.0-5-g8c1d971+7e5b4c34a6/lib/libgeom.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/geom/21.0.0-5-g8c1d971+7e5b4c34a6/lib/libgeom.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_base/21.0.0-6-g1930a60+b18c5f6b76/lib/libdaf_base.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_base/21.0.0-6-g1930a60+b18c5f6b76/lib/libdaf_base.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_base/21.0.0-6-g1930a60+b18c5f6b76/lib/libdaf_base.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_exceptions/21.0.0-2-gde069b7+4f46bdaea8/lib/libpex_exceptions.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_exceptions/21.0.0-2-gde069b7+4f46bdaea8/lib/libpex_exceptions.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/log/21.0.0-5-gcc89fd6+b18c5f6b76/lib/liblog.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/log/21.0.0-5-gcc89fd6+b18c5f6b76/lib/liblog.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/log/21.0.0-5-gcc89fd6+b18c5f6b76/lib/liblog.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/astshim/21.0.0-2-g45278ab+64c1bc5aa5/lib/libastshim.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/astshim/21.0.0-2-g45278ab+64c1bc5aa5/lib/libastshim.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/astshim/21.0.0-2-g45278ab+64c1bc5aa5/lib/libastshim.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/astshim/21.0.0-2-g45278ab+64c1bc5aa5/lib/libastshim.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/utils/21.0.0-7-g0503b2e+18535a8d22/lib/libutils.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/utils/21.0.0-7-g0503b2e+18535a8d22/lib/libutils.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/base/21.0.0-7-gdf92d54+64c1bc5aa5/lib/libbase.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/base/21.0.0-7-gdf92d54+64c1bc5aa5/lib/libbase.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/sphgeom/21.0.0-1-g8760c09+64c1bc5aa5/lib/libsphgeom.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/sphgeom/21.0.0-1-g8760c09+64c1bc5aa5/lib/libsphgeom.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/sphgeom/21.0.0-1-g8760c09+64c1bc5aa5/lib/libsphgeom.so)

linux-vdso.so.1 => (0x00002b0c6607c000)

libgalsim.so.2.2 => not found

libmeas_base.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-14-g3bd782b+16b33694c2/lib/libmeas_base.so (0x00002b0c660be000)

libafw.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-46-g27369044f+532d44eec1/lib/libafw.so (0x00002b0c6627e000)

libgeom.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/geom/21.0.0-5-g8c1d971+7e5b4c34a6/lib/libgeom.so (0x00002b0c66185000)

libdaf_base.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_base/21.0.0-6-g1930a60+b18c5f6b76/lib/libdaf_base.so (0x00002b0c661cb000)

libpex_exceptions.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_exceptions/21.0.0-2-gde069b7+4f46bdaea8/lib/libpex_exceptions.so (0x00002b0c66265000)

libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00002b0c66d64000)

libm.so.6 => /lib64/libm.so.6 (0x00002b0c6706c000)

libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b0c6736e000)

libc.so.6 => /lib64/libc.so.6 (0x00002b0c67584000)

liblog.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/log/21.0.0-5-gcc89fd6+b18c5f6b76/lib/liblog.so (0x00002b0c6626e000)

liblog4cxx.so.11 => not found

libfftw3.so.3 => not found

libastshim.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/astshim/21.0.0-2-g45278ab+64c1bc5aa5/lib/libastshim.so (0x00002b0c67952000)

libast.so.9 => not found

libcfitsio.so.8 => not found

libgsl.so.25 => not found

libopenblas.so.0 => not found

libMinuit2.so => not found

liblog4cxx.so.11 => not found

libutils.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/utils/21.0.0-7-g0503b2e+18535a8d22/lib/libutils.so (0x00002b0c679ae000)

libbase.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/base/21.0.0-7-gdf92d54+64c1bc5aa5/lib/libbase.so (0x00002b0c679ed000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b0c679f9000)

libsphgeom.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/sphgeom/21.0.0-1-g8760c09+64c1bc5aa5/lib/libsphgeom.so (0x00002b0c67c16000)

/lib64/ld-linux-x86-64.so.2 (0x00002b0c6605a000)

liblog4cxx.so.11 => not found

libast.so.9 => not found

libast_grf_2.0.so.9 => not found

libast_grf_3.2.so.9 => not found

libast_grf_5.6.so.9 => not found

libast_grf3d.so.9 => not found

libast_err.so.9 => not found

libdl.so.2 => /lib64/libdl.so.2 (0x00002b0c67c55000)

So, there have been times where I had to disable the intel MPI/MKL library to run a pipetask run properly. If I enable the intel MPI/MKL library and load Python 3.8.2 as well, I get a different result for ldd $EUPS_PATH/Linux64/meas_extensions_shapeHSM/*/lib/libmeas_extensions_shapeHSM.so, that is:

(lsst-scipipe-0.6.0) [jahumada@leftraru3 jahumada]$ ldd $EUPS_PATH/Linux64/meas_extensions_shapeHSM/*/lib/libmeas_extensions_shapeHSM.so

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /home/lmod/software/Core/GCCcore/8.2.0/lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-46-g27369044f+532d44eec1/lib/libafw.so)

/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-5-g810ae3f+b39e0bca51/lib/libmeas_extensions_shapeHSM.so: /home/lmod/software/Core/GCCcore/8.2.0/lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/astshim/21.0.0-2-g45278ab+64c1bc5aa5/lib/libastshim.so)

linux-vdso.so.1 => (0x00002b45157bc000)

libgalsim.so.2.2 => not found

libmeas_base.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-14-g3bd782b+16b33694c2/lib/libmeas_base.so (0x00002b4515800000)

libafw.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/afw/21.0.0-46-g27369044f+532d44eec1/lib/libafw.so (0x00002b45159be000)

libgeom.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/geom/21.0.0-5-g8c1d971+7e5b4c34a6/lib/libgeom.so (0x00002b45158c7000)

libdaf_base.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_base/21.0.0-6-g1930a60+b18c5f6b76/lib/libdaf_base.so (0x00002b451590d000)

libpex_exceptions.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pex_exceptions/21.0.0-2-gde069b7+4f46bdaea8/lib/libpex_exceptions.so (0x00002b45159a7000)

libstdc++.so.6 => /home/lmod/software/Core/GCCcore/8.2.0/lib64/libstdc++.so.6 (0x00002b45164a4000)

libm.so.6 => /lib64/libm.so.6 (0x00002b451663e000)

libgcc_s.so.1 => /home/lmod/software/Core/GCCcore/8.2.0/lib64/libgcc_s.so.1 (0x00002b4516940000)

libc.so.6 => /lib64/libc.so.6 (0x00002b4516959000)

liblog.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/log/21.0.0-5-gcc89fd6+b18c5f6b76/lib/liblog.so (0x00002b45159b0000)

liblog4cxx.so.11 => not found

libfftw3.so.3 => not found

libastshim.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/astshim/21.0.0-2-g45278ab+64c1bc5aa5/lib/libastshim.so (0x00002b4516d28000)

libast.so.9 => not found

libcfitsio.so.8 => not found

libgsl.so.25 => not found

libopenblas.so.0 => not found

libMinuit2.so => not found

liblog4cxx.so.11 => not found

libutils.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/utils/21.0.0-7-g0503b2e+18535a8d22/lib/libutils.so (0x00002b4516d85000)

libbase.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/base/21.0.0-7-gdf92d54+64c1bc5aa5/lib/libbase.so (0x00002b4516dc4000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b4516dd0000)

libsphgeom.so => /mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/sphgeom/21.0.0-1-g8760c09+64c1bc5aa5/lib/libsphgeom.so (0x00002b4516fed000)

/lib64/ld-linux-x86-64.so.2 (0x00002b451579a000)

liblog4cxx.so.11 => not found

libast.so.9 => not found

libast_grf_2.0.so.9 => not found

libast_grf_3.2.so.9 => not found

libast_grf_5.6.so.9 => not found

libast_grf3d.so.9 => not found

libast_err.so.9 => not found

libdl.so.2 => /lib64/libdl.so.2 (0x00002b451702c000)

Could these not found extensions be because I am using a weekly (27th version) of the 21st version of the pipeline?

OK, you’ll need to reinstall rubin-env in your lsst-scipipe-0.6.0 environment. The weekly you are using was not certified or built to use galsim 2.3. We pinned the version in the environment to handle this, but it doesn’t automatically update previously-installed environments. I believe the minimum you can do to get going again is
$ conda install galsim=2.2
You could try instead
$ conda install rubin-env=0.6.0=ha770c72_4
which will be safer, as it also constrains some other packages.
I don’t think you can do
$ conda upgrade rubin-env=0.6.0
as that may not downgrade existing packages.

1 Like

Thanks ktl! I ran conda install rubin-env=0.6.0=ha770c72_4 and it installed succesfully.
Now I do get the error of the missing reference catalog

(lsst-scipipe-0.6.0) [jahumada@leftraru4 jahumada]$ pipetask run -b data_hits/ -p $AP_PIPE_DIR/pipelines/DarkEnergyCamera/ProcessCcd.yaml -i DECam/raw/all,hits_master_calib/on20140228_bias,hits_master_calib/on20140228_flats --instrument lsst.obs.decam.DarkEnergyCamera --output-run processCcdOutputs/pon20140228
Traceback (most recent call last):
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/pipeline.py", line 711, in makeDatasetTypesSet
    datasetType = registry.getDatasetType(c.name)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-109-g75b85349+7e5b4c34a6/python/lsst/daf/butler/registries/sql.py", line 391, in getDatasetType
    return self._managers.datasets[name].datasetType
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-109-g75b85349+7e5b4c34a6/python/lsst/daf/butler/registry/interfaces/_datasets.py", line 491, in __getitem__
    raise KeyError(f"Dataset type with name '{name}' not found.")
KeyError: "Dataset type with name 'ps1_pv3_3pi_20170110' not found."

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/bin/pipetask", line 29, in <module>
    sys.exit(main())
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cli/pipetask.py", line 43, in main
    return cli()
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/jahumada/.conda/envs/lsst-scipipe-0.6.0/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 102, in run
    qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 145, in qgraph
    qgraph = f.makeGraph(pipelineObj, args)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-34-g2cbe45c+f487817123/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 550, in makeGraph
    qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query, metadata=metadata)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/graphBuilder.py", line 892, in makeGraph
    scaffolding = _PipelineScaffolding(pipeline, registry=self.registry)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/graphBuilder.py", line 411, in __init__
    datasetTypes = PipelineDatasetTypes.fromPipeline(pipeline, registry=registry)
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/pipeline.py", line 911, in fromPipeline
    thisTask = TaskDatasetTypes.fromTaskDef(
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/pipeline.py", line 784, in fromTaskDef
    prerequisites=makeDatasetTypesSet("prerequisiteInputs"),
  File "/mnt/flock/jahumada/lsstw_v21_27/lsst-w.2021.27/scripts/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-29-gf27b79e+8c09b75fa7/python/lsst/pipe/base/pipeline.py", line 713, in makeDatasetTypesSet
    raise LookupError(
LookupError: DatasetType 'ps1_pv3_3pi_20170110' referenced by CalibrateConnections uses 'skypix' as a dimension placeholder, but does not already exist in the registry.  Note that reference catalog names are now used as the dataset type name instead of 'ref_cat'. 

From what I’ve read, the only way to get a gen3 refcat is to ingest it into a gen2 repo and then convert it to gen3 (https://jira.lsstcorp.org/browse/DM-29543). There is an example that uses Gaia DR2 How to generate an LSST reference catalog — LSST Science Pipelines but I would like to use Pan-STARRS to construct the reference catalog. I would like to know if you have any recommendations as to what procedures would be best for me to do? Maybe I could reproduce the Gaia DR2 example with Pan-STARRS?

Thanks a lot for your time :slight_smile:

If you can wait, I’m preparing instructions on how to transfer an existing gen2 refcat (we have a PS1 refcat in the LSST format) to a gen3 repo. It should only be a couple of commands. See DM-30624 for updates.

Converting an external refcat to the LSST format and then ingesting it into a gen3 repo is work I’m doing currently; DM-29543 is the right ticket to follow that.

1 Like

Okay :slight_smile: I will go ahead and download the images that work for me from here https://tigress-web.princeton.edu/~pprice/ps1_pv3_3pi_20170110/

I will keep an eye on the tickets you mention as well!

Hi John :slight_smile: I was able to download the Pan-STARRS images needed to calibrate my images :smiley: .
From what I’ve checked, there are no updates published on the tickets DM-30624 and DM-29543 for now :open_mouth: . I was wondering if you had any methods/commands I could try in the meantime? I would like to try stuff :slight_smile: .
I actually don’t dispose of a lot of time to wait, so any news is great news for me.
Thanks a lot for your time!

I’ve made some progress on this. You can try the following commands, where REPO is the path to your gen3 butler repo. Note that you’ll need a pretty recent pipelines version, as butler register-dataset-type was added at the end of August.

  1. Create an astropy-readable table file containing one row per input file in the PS1 refcat, with two columns: the filename and htm7-pixel index. I think something like this code will get you the right output:
import glob
import astropy.table
table = astropy.table.Table(names=("filename", "htm7"), dtype=('str', 'int'))
for file in glob.glob("ps1_pv3_3pi_20170110/*.fits"):
    table.add_row((file, int(splitext(file)[0])))
table.write("filename_to_htm.ecsv")
  1. Register the new ps1 refcat dataset:
butler register-dataset-type REPO ps1_pv3_3pi_20170110 SimpleCatalog htm7
  1. Ingest the LSST-formatted files you downloaded:
butler ingest-files -t direct REPO ps1_pv3_3pi_20170110 refcats filename_to_htm.ecsv

The ingested data should appear via butler query-datasets --collections refcats REPO.

1 Like

Hello,

I’m having problems reducing DECam data. In particular the bias combine task is giving tons of errors and producing very few master frames. Below I enclose some of the pipeline output (which is a re-run):

ValueError: Failure from formatter ‘lsst.obs.base.formatters.fitsExposure.FitsExposureFormatter’ for dataset d79309f0-85bf-4569-9b78-b177483c4bfe (cpBiasProc from file:///mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_
20211028T155954Z.fits):
File “src/fits.cc”, line 1578, in lsst::afw::fits::Fits::Fits(const string&, const string&, int)
cfitsio error: error reading from FITS file (108) : Opening file ‘/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fits’ with mode ‘r’
cfitsio error stack:
Error reading data buffer from file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fit
s
Failed to find the END keyword in header (ffgphd).
Failed to read the required primary array header keywords.
ffopen could not interpret primary array header of file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fit
s
{0}
lsst::afw::fits::FitsError: 'cfitsio error: error reading from FITS file (108) : Opening file ‘/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fits’ with mode ‘r’
cfitsio error stack:
Error reading data buffer from file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fit
s
Failed to find the END keyword in header (ffgphd).
Failed to read the required primary array header keywords.
ffopen could not interpret primary array header of file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191226/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191226_S14_master_calib_on20191122_bias_20211028T155954Z.fit
s

ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(CalibCombineTask, label=cpBiasCombine) dataId={instrument: ‘DECam’, detector: 13}> failed; processing will continue for remaining tasks.
ctrl.mpexec.mpGraphExecutor INFO: Executed 0 quanta successfully, 13 failed and 44 remain out of total 57 quanta.
timer.lsst.daf.butler.datastores.fileDatastore ERROR: Reading from location file:///mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits with formatter lsst.obs.base.formatters.fitsExposure.FitsExposure
Formatter: Took 17.8089 seconds
ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task ‘cpBiasCombine’ on quantum {instrument: ‘DECam’, detector: 26} failed
Traceback (most recent call last):
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/daf_butler/22.0.1-70-gc781c74d+f4bf734fca/python/lsst/daf/butler/datastores/fileDatastore.py”, line 1215, in _read_artifact_into_memory
result = formatter.read(component=getInfo.component if isComponent else None)
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-18-g3ad3863+50f8e256e0/python/lsst/obs/base/formatters/fitsExposure.py”, line 89, in read
return self.readFull()
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-18-g3ad3863+50f8e256e0/python/lsst/obs/base/formatters/fitsExposure.py”, line 560, in readFull
result = self.reader.read(self.checked_parameters)
lsst.pex.exceptions.wrappers.FitsError:
File “src/fits.cc”, line 1419, in void lsst::afw::fits::Fits::readImageImpl(int, T
, long int
, long int*, long int*) [with T = float]
cfitsio error (/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits): error reading from FITS file (108) : Reading image
cfitsio error stack:
Error reading data buffer from file:
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/2021102
8T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0
_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits
Error reading elements 1 thru 2813 from column 1 (ffgclb).
error reading compressed byte stream from binary table
{0}
lsst::afw::fits::FitsError: 'cfitsio error (/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits): error reading from FITS file (108) : Reading image
(base) [jahumada@leftraru4 ~]$ ls /mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits*
/mnt/flock/jahumada/eridanus_data/processed/master_calib/on20191122_bias/20211028T155954Z/cpBiasProc/20191122/ct4m20191122t191052/cpBiasProc_DECam_solid_plate_0_0_0_0_ct4m20191122t191052_S2_master_calib_on20191122_bias_20211028T155954Z.fits
(base) [jahumada@leftraru4 ~]$ tail plira_23665563.err
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-18-gee15f3f+b3a3f94acb/python/lsst/ctrl/mpexec/cli/script/run.py”, line 171, in run
f.runPipeline(qgraphObj, taskFactory, args)
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-18-gee15f3f+b3a3f94acb/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 669, in runPipeline
executor.execute(graph, butler)
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-18-gee15f3f+b3a3f94acb/python/lsst/ctrl/mpexec/mpGraphExecutor.py”, line 300, in execute
self._executeQuantaMP(graph, butler)
File “/mnt/flock/jahumada/lsstw_v21_37/lsst-w.2021.37/scripts/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/ctrl_mpexec/22.0.1-18-gee15f3f+b3a3f94acb/python/lsst/ctrl/mpexec/mpGraphExecutor.py”, line 486, in _executeQuantaMP
raise MPGraphExecutorError(“One or more tasks failed or timed out during execution.”)
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed or timed out during execution.
slurmstepd: error: Detected 48 oom-kill event(s) in step 23665563.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

Sorry about so much mess!

Paulina.

This looks like the file is completely corrupt. Can other tools read it? We usually only encounter problems like this if you’ve run out of disk space.