Obs_decam + processCcd.py segfault

Hello,

I recently built a version of DMstack from a recent weekly release, and also installed obs_decam. When I run obs_decam’s processCcd on some CP-calibrated DECam data, I get a segfault, though an icSrc.fits file is created. Per the suggestion of some experts (@ctslater,@rbliu, @jbosch), I am pasting my whole process here. Apologies in advance for the walls o’ text…

I should note that I installed a weekly release, rather than v14_0 (still listed as the “standard” on pipelines.lsst.io website) because obs_decam would not install with v14_0.

I downloaded a master version of newinstall.sh, and followed the instructions on pipelines.lsst.io/install/newinstall.html:

curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh
bash newinstall.sh -ct

source loadLSST.bash
eups distrib install -t w_2017_48 lsst_distrib
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
setup lsst_distrib

And all proceeds without too much fuss. To install obs_decam, I followed the instructions here https://github.com/lsst/obs_decam/blob/master/README.md:

$ cd /Users/jemccleary/Research
$ git clone git://github.com/lsst/obs_decam.git
$ cd obs_decam
$ setup -t w_2017_48 -r .
$ scons install declare --tag=current

I didn’t notice any special error messages. Putting this to work on some DECam Abell 1348 data from the NOAO archive,

$ ingestImagesDecam.py /Users/jmcclear/Research/A1348 --mode=link cpCal/instcal/*
root INFO: Loading config overrride file '/Users/jmcclear/Research/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/obs_decam/14.0-4-g3127c16+22/config/ingest.py’
CameraMapper INFO: Loading Posix exposure registry from /Users/jmcclear/Research/A1348
ingest.parse WARN: Unable to find value for ccdnum (derived from CCDNUM)
ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM)
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/instcal/c4d_130204_063637_ooi_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175045/instcal0175045.fits.fz
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/dqmask/c4d_130204_063637_ood_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175045/dqmask0175045.fits.fz
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/wtmap/c4d_130204_063637_oow_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175045/wtmap0175045.fits.fz
ingest.parse WARN: Unable to find value for ccdnum (derived from CCDNUM)
ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM)
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/instcal/c4d_130204_064309_ooi_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175046/instcal0175046.fits.fz
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/dqmask/c4d_130204_064309_ood_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175046/dqmask0175046.fits.fz
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/wtmap/c4d_130204_064309_oow_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175046/wtmap0175046.fits.fz

I didn’t use the config override file, perhaps that’s the problem… EDIT nope, IngestConfig object has no ‘isr’ attribute! nvm.

Anyway, going to the processCcd step,

$processCcd.py /Users/jmcclear/Research/A1348 --id visit=0175045 ccdnum=18 --output /Users/jmcclear/Research/A1348/outDir/ -C override_config.py --config calibrate.doAstrometry=False calibrate.doPhotoCal=False
root INFO: Loading config overrride file '/Users/jmcclear/Research/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/obs_decam/14.0-4-g3127c16+22/config/processCcd.py’
CameraMapper INFO: Loading exposure registry from /Users/jmcclear/Research/A1348/registry.sqlite3
makeWcs WARN: Stripping PVi_j keys from projection RA—TPV/DEC–TPV
root INFO: Running: /Users/jmcclear/Research/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/pipe_tasks/14.0-27-g4464a260+2/bin/processCcd.py /Users/jmcclear/Research/A1348 --id visit=0175045 ccdnum=18 --output /Users/jmcclear/Research/A1348/outDir/ -C override_config.py --config calibrate.doAstrometry=False calibrate.doPhotoCal=False
processCcd INFO: Processing {‘visit’: 175045, ‘ccdnum’: 18, ‘hdu’: 18}
processCcd.isr INFO: Loading DECam community pipeline file {‘visit’: 175045, ‘ccdnum’: 18, ‘hdu’: 18}
makeWcs WARN: Stripping PVi_j keys from projection RA—TPV/DEC–TPV
processCcd.charImage INFO: Processing {‘visit’: 175045, ‘ccdnum’: 18, ‘hdu’: 18}
processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processCcd.charImage.repair INFO: Identified 7 cosmic rays.
processCcd.charImage.detection INFO: Detected 689 positive sources to 50 sigma.
processCcd.charImage.detection INFO: Resubtracting the background after object detection
processCcd.charImage.measurement INFO: Measuring 689 sources (689 parents, 0 children)
processCcd.charImage.measurePsf INFO: Measuring PSF
processCcd.charImage.measurePsf.reserve INFO: Reserved 0/154 sources
processCcd.charImage.measurePsf INFO: PSF star selector found 154 candidates
processCcd.charImage.measurePsf INFO: PSF determination using 117/154 stars.
processCcd.charImage INFO: iter 1; PSF sigma=1.37, dimensions=(25, 25); median background=2865.96
processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processCcd.charImage.repair INFO: Identified 7 cosmic rays.
processCcd.charImage.detection INFO: Detected 718 positive sources to 50 sigma.
processCcd.charImage.detection INFO: Resubtracting the background after object detection
processCcd.charImage.measurement INFO: Measuring 718 sources (718 parents, 0 children)
processCcd.charImage.measurePsf INFO: Measuring PSF
processCcd.charImage.measurePsf.reserve INFO: Reserved 0/154 sources
processCcd.charImage.measurePsf INFO: PSF star selector found 154 candidates
processCcd.charImage.measurePsf INFO: PSF determination using 99/154 stars.
processCcd.charImage INFO: iter 2; PSF sigma=1.37, dimensions=(25, 25); median background=2866.00
processCcd.charImage.repair INFO: Identified 6 cosmic rays.
processCcd.charImage.measurement INFO: Measuring 718 sources (718 parents, 0 children)
processCcd.charImage.measureApCorr INFO: Measuring aperture corrections for 2 flux fields
processCcd.charImage.measureApCorr INFO: Aperture correction for base_GaussianFlux: RMS 0.025656 from 98
processCcd.charImage.measureApCorr INFO: Aperture correction for base_PsfFlux: RMS 0.023181 from 96
processCcd.charImage.applyApCorr INFO: Applying aperture corrections to 2 flux fields
processCcd.calibrate INFO: Processing {‘visit’: 175045, ‘ccdnum’: 18, ‘hdu’: 18}
processCcd.calibrate.detection INFO: Detected 3788 positive sources to 5 sigma.
processCcd.calibrate.detection INFO: Resubtracting the background after object detection
processCcd.calibrate.deblend INFO: Deblending 3788 sources
processCcd.calibrate.deblend INFO: Deblended: of 3788 sources, 644 were deblended, creating 2751 children, total 6539 sources
processCcd.calibrate.measurement INFO: Measuring 6539 sources (3788 parents, 2751 children)
Caught signal 11, backtrace follows:
0 libutils.dylib 0x0000000106a1c5b1 lsst::utils::(anonymous namespace)::signalHandler(int) + 81
1 libsystem_platform.dylib 0x00007fff7606af5a (null) + 26
1 libsystem_platform.dylib 0x00007fff7606af5a _sigtramp + 26
2 applyLookupTable.so 0x000000010328373d pybind11_object_dealloc + 605
3 _heavyFootprint.so 0x0000000128ec084e lsst::afw::detection::HeavyFootprint<float, int, float>::~HeavyFootprint() + 46
4 libc++.1.dylib 0x00007fff73e598fd std::__1::__shared_weak_count::__release_shared() + 43
5 applyLookupTable.so 0x000000010328350b pybind11_object_dealloc + 43
6 libpython3.6m.dylib 0x000000010089e604 frame_dealloc + 132
7 libpython3.6m.dylib 0x0000000100942bb8 fast_function + 408
8 libpython3.6m.dylib 0x0000000100941901 call_function + 401
9 libpython3.6m.dylib 0x000000010093e0d2 (null) + 27362
9 libpython3.6m.dylib 0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
10 libpython3.6m.dylib 0x000000010094238f (null) + 2447
10 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
11 libpython3.6m.dylib 0x0000000100942f42 (null) + 738
11 libpython3.6m.dylib 0x0000000100942f42 _PyFunction_FastCallDict + 738
12 libpython3.6m.dylib 0x0000000100876e87 (null) + 247
12 libpython3.6m.dylib 0x0000000100876e87 _PyObject_FastCallDict + 247
13 libpython3.6m.dylib 0x0000000100876fa5 (null) + 149
13 libpython3.6m.dylib 0x0000000100876fa5 _PyObject_Call_Prepend + 149
14 libpython3.6m.dylib 0x0000000100876cc0 PyObject_Call + 96
15 libpython3.6m.dylib 0x000000010093e3b5 (null) + 28101
15 libpython3.6m.dylib 0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
16 libpython3.6m.dylib 0x000000010094238f (null) + 2447
16 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
17 libpython3.6m.dylib 0x0000000100942f42 (null) + 738
17 libpython3.6m.dylib 0x0000000100942f42 _PyFunction_FastCallDict + 738
18 libpython3.6m.dylib 0x0000000100876e87 (null) + 247
18 libpython3.6m.dylib 0x0000000100876e87 _PyObject_FastCallDict + 247
19 libpython3.6m.dylib 0x0000000100876fa5 (null) + 149
19 libpython3.6m.dylib 0x0000000100876fa5 _PyObject_Call_Prepend + 149
20 libpython3.6m.dylib 0x0000000100876cc0 PyObject_Call + 96
21 libpython3.6m.dylib 0x000000010093e3b5 (null) + 28101
21 libpython3.6m.dylib 0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
22 libpython3.6m.dylib 0x000000010094238f (null) + 2447
22 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
23 libpython3.6m.dylib 0x0000000100942c41 fast_function + 545
24 libpython3.6m.dylib 0x0000000100941901 call_function + 401
25 libpython3.6m.dylib 0x000000010093e171 (null) + 27521
25 libpython3.6m.dylib 0x000000010093e171 _PyEval_EvalFrameDefault + 27521
26 libpython3.6m.dylib 0x000000010094238f (null) + 2447
26 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
27 libpython3.6m.dylib 0x0000000100942c41 fast_function + 545
28 libpython3.6m.dylib 0x0000000100941901 call_function + 401
29 libpython3.6m.dylib 0x000000010093e0d2 (null) + 27362
29 libpython3.6m.dylib 0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
30 libpython3.6m.dylib 0x000000010094238f (null) + 2447
30 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
31 libpython3.6m.dylib 0x00000001009375a4 PyEval_EvalCodeEx + 100
32 libpython3.6m.dylib 0x00000001008a02ad function_call + 381
33 libpython3.6m.dylib 0x0000000100876cc0 PyObject_Call + 96
34 libpython3.6m.dylib 0x000000010093e3b5 (null) + 28101
34 libpython3.6m.dylib 0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
35 libpython3.6m.dylib 0x000000010094238f (null) + 2447
35 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
36 libpython3.6m.dylib 0x0000000100942c41 fast_function + 545
37 libpython3.6m.dylib 0x0000000100941901 call_function + 401
38 libpython3.6m.dylib 0x000000010093e171 (null) + 27521
38 libpython3.6m.dylib 0x000000010093e171 _PyEval_EvalFrameDefault + 27521
39 libpython3.6m.dylib 0x000000010094238f (null) + 2447
39 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
40 libpython3.6m.dylib 0x0000000100942c41 fast_function + 545
41 libpython3.6m.dylib 0x0000000100941901 call_function + 401
42 libpython3.6m.dylib 0x000000010093e171 (null) + 27521
42 libpython3.6m.dylib 0x000000010093e171 _PyEval_EvalFrameDefault + 27521
43 libpython3.6m.dylib 0x000000010094238f (null) + 2447
43 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
44 libpython3.6m.dylib 0x00000001009375a4 PyEval_EvalCodeEx + 100
45 libpython3.6m.dylib 0x00000001008a02ad function_call + 381
46 libpython3.6m.dylib 0x0000000100876cc0 PyObject_Call + 96
47 libpython3.6m.dylib 0x000000010093e3b5 (null) + 28101
47 libpython3.6m.dylib 0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
48 libpython3.6m.dylib 0x000000010094238f (null) + 2447
48 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
49 libpython3.6m.dylib 0x0000000100942c41 fast_function + 545
50 libpython3.6m.dylib 0x0000000100941901 call_function + 401
51 libpython3.6m.dylib 0x000000010093e171 (null) + 27521
51 libpython3.6m.dylib 0x000000010093e171 _PyEval_EvalFrameDefault + 27521
52 libpython3.6m.dylib 0x000000010094238f (null) + 2447
52 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
53 libpython3.6m.dylib 0x00000001009375a4 PyEval_EvalCodeEx + 100
54 libpython3.6m.dylib 0x00000001008a02ad function_call + 381
55 libpython3.6m.dylib 0x0000000100876cc0 PyObject_Call + 96
56 libpython3.6m.dylib 0x000000010093e3b5 (null) + 28101
56 libpython3.6m.dylib 0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
57 libpython3.6m.dylib 0x000000010094238f (null) + 2447
57 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
58 libpython3.6m.dylib 0x0000000100942f42 (null) + 738
58 libpython3.6m.dylib 0x0000000100942f42 _PyFunction_FastCallDict + 738
59 libpython3.6m.dylib 0x0000000100876e87 (null) + 247
59 libpython3.6m.dylib 0x0000000100876e87 _PyObject_FastCallDict + 247
60 libpython3.6m.dylib 0x0000000100876fa5 (null) + 149
60 libpython3.6m.dylib 0x0000000100876fa5 _PyObject_Call_Prepend + 149
61 libpython3.6m.dylib 0x0000000100876cc0 PyObject_Call + 96
62 libpython3.6m.dylib 0x000000010093e3b5 (null) + 28101
62 libpython3.6m.dylib 0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
63 libpython3.6m.dylib 0x0000000100942dbc (null) + 348
63 libpython3.6m.dylib 0x0000000100942dbc _PyFunction_FastCallDict + 348
64 libpython3.6m.dylib 0x0000000100876e87 (null) + 247
64 libpython3.6m.dylib 0x0000000100876e87 _PyObject_FastCallDict + 247
65 libpython3.6m.dylib 0x0000000100876fa5 (null) + 149
65 libpython3.6m.dylib 0x0000000100876fa5 _PyObject_Call_Prepend + 149
66 libpython3.6m.dylib 0x0000000100876cc0 PyObject_Call + 96
67 libpython3.6m.dylib 0x00000001008d6e15 slot_tp_call + 117
68 libpython3.6m.dylib 0x0000000100876e55 (null) + 197
68 libpython3.6m.dylib 0x0000000100876e55 _PyObject_FastCallDict + 197
69 libpython3.6m.dylib 0x000000010093323a map_next + 154
70 libpython3.6m.dylib 0x00000001008a22da listextend + 666
71 libpython3.6m.dylib 0x00000001008a3723 list_init + 163
72 libpython3.6m.dylib 0x00000001008d3eb9 type_call + 313
73 libpython3.6m.dylib 0x0000000100876e55 (null) + 197
73 libpython3.6m.dylib 0x0000000100876e55 _PyObject_FastCallDict + 197
74 libpython3.6m.dylib 0x0000000100941828 call_function + 184
75 libpython3.6m.dylib 0x000000010093e0d2 (null) + 27362
75 libpython3.6m.dylib 0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
76 libpython3.6m.dylib 0x0000000100942b9d fast_function + 381
77 libpython3.6m.dylib 0x0000000100941901 call_function + 401
78 libpython3.6m.dylib 0x000000010093e0d2 (null) + 27362
78 libpython3.6m.dylib 0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
79 libpython3.6m.dylib 0x000000010094238f (null) + 2447
79 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
80 libpython3.6m.dylib 0x0000000100942c41 fast_function + 545
81 libpython3.6m.dylib 0x0000000100941901 call_function + 401
82 libpython3.6m.dylib 0x000000010093e0d2 (null) + 27362
82 libpython3.6m.dylib 0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
83 libpython3.6m.dylib 0x000000010094238f (null) + 2447
83 libpython3.6m.dylib 0x000000010094238f _PyEval_EvalCodeWithName + 2447
84 libpython3.6m.dylib 0x0000000100937534 PyEval_EvalCode + 100
85 libpython3.6m.dylib 0x000000010096d511 PyRun_FileExFlags + 209
86 libpython3.6m.dylib 0x000000010096ccc3 PyRun_SimpleFileExFlags + 851
87 libpython3.6m.dylib 0x0000000100985c49 Py_Main + 3529
88 python 0x0000000100866df8 main + 232
89 libdyld.dylib 0x00007fff75dea145 start + 1
90 ??? 0x000000000000000d 0x0 + 13
Segmentation fault: 11

Would you obtain the same error if you use the obs_decam of your lsst_distrib installation instead of the local one? In other words, is there any differences if you skip everything in your second code block? (obs_decam is part of lsst_distrib)

Yes, processCcd.py (calibrate.py as well) should be always located in pipe_tasks instead of obs_*. It is a good point to repeat the error only with setup lsst_distrib (but no setup obs_decam separately) before you run ingestImagesDecam.py and processCcd.py.
I have obs_decam included in a default v13.0 installation. So I guess it should still be there for a v14.0 version.

Hi all,

Thanks for the good suggestions, but unfortunately nothing has worked so far. I was going to try reducing the DECam data from scratch (i.e., ingesting calibration files) to see if that made a difference, but that seems to not be working at the moment. At least, it’s not clear to me what’s missing from the documentation here: https://github.com/lsst/obs_decam. The ingestCalibs can’t find a CALIB_ID on the flat fields I downloaded from portal-nvo.noao.edu, it also doesn’t know what a “fringe file” is, and those

ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM)

warnings still persist. I’m not sure if any of this is related to the segfault above, but could it be? Is there some new processing on the DECam CP end that’s not recognized by existing obs_decam? I know PLVER gets updated periodically…

So I think Jim’s comment in the previous thread still applies here; our prior about segfaults is that they most frequently come from some mismatch of various library versions. This backtrace is also very suspicious in that it mentions applyLookupTable.so, which is completely unrelated to measurement (where the crash occurs) and which should never be called given that you’re using community pipeline data (applyLookupTable is for linearity correction). This makes it hard to tell if the backtrace is leading us to the problem or just reflects some sort of corruption issue.

So there are a few options. One is to hunt down this issue further, which I think means increasing the log level so to DEBUG or TRACE so that we can pin down where exactly the segfault happens. It would be good to double check your compiler versions too. I don’t really know if we will be able to get to a good answer that way, but it would at least be a prerequisite. Another option is to delete and reinstall the stack, on the theory that something is either damaged or misinstalled (it sounds like you did all the right steps though). An interesting option might be to try the stack Docker images, since they will completely isolate the environment the stack sees from anything else you have installed. If the docker images still crash, then we have a serious problem!

Hi all,

here’s the part where I have to cover myself with shame. Clicking around on the Docker page that @ctslater linked led me here: https://pipelines.lsst.io/install/package-development.html. It turns out that without running

setup obs_decam

after setting up the DMstack environment as follows,

source loadLSST.bash
setup lsst_distrib

the "eups list -s " command showed that I was still running the lsst_distrib version of obs_decam, not the updated version available from GitHub. Simply sourcing this accurately fixed the problem! I now have beautiful src.fits files.

I’m really sorry to have bothered everyone with something that now appears so trivial!

Perhaps for the sake of any other offworlders joining the party and to prevent future wastes of time for you, it might be useful to update the documentation both for obs_decam and for lsst.io. Perhaps this is obvious to everyone, but perhaps not. We could state that you (a) have to open a new terminal window, as I found that trying to run obs_decam fresh off the install leads to the same error, and (b) you have to explicitly source any new packages (i.e. obs_decam) after sourcing lsst_distrib. If it would be helpful, I’d be happy to do this myself!

In any case, thank you all so much for your time and help!

Thanks for your input @jmac. If you’d like to suggest changes, you can open a GitHub Pull Request at https://github.com/lsst/pipelines_lsst_io and I’ll help you merge those in. Otherwise, I can work on that documentation and get feedback from you on whether we’re doing better.

Edit: Looking at the thread, another thing we could improve is showing how to install different cameras, like obs_decam, without having to build it from source. That way you could get the v14 version of obs_decam alongside the v14 version of the rest of the stack. I’ll look into this.

Hi Jonathan!

I’d love to actually contribute something to the project instead of just being deadweight, so I’ll go ahead with the pull request and try to merge in with the existing documentation (if that’s the right place to do it).

Having a section on how to install different cameras would be useful too, since my group’s medium-term goal is to write up an “obs_superbit” module (https://sites.physics.utoronto.ca/bit) and, in time, make it available as well!

Perfect! Glad it worked.