Obs_decam + processCcd.py segfault

jmac · December 6, 2017, 9:18pm

Hello,

I recently built a version of DMstack from a recent weekly release, and also installed obs_decam. When I run obs_decam’s processCcd on some CP-calibrated DECam data, I get a segfault, though an icSrc.fits file is created. Per the suggestion of some experts (@ctslater,@rbliu, @jbosch), I am pasting my whole process here. Apologies in advance for the walls o’ text…

I should note that I installed a weekly release, rather than v14_0 (still listed as the “standard” on pipelines.lsst.io website) because obs_decam would not install with v14_0.

I downloaded a master version of newinstall.sh, and followed the instructions on pipelines.lsst.io/install/newinstall.html:

curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh
bash newinstall.sh -ct

source loadLSST.bash
eups distrib install -t w_2017_48 lsst_distrib
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
setup lsst_distrib

And all proceeds without too much fuss. To install obs_decam, I followed the instructions here https://github.com/lsst/obs_decam/blob/master/README.md:

$ cd /Users/jemccleary/Research
$ git clone git://github.com/lsst/obs_decam.git
$ cd obs_decam
$ setup -t w_2017_48 -r .
$ scons install declare --tag=current

I didn’t notice any special error messages. Putting this to work on some DECam Abell 1348 data from the NOAO archive,

$ ingestImagesDecam.py /Users/jmcclear/Research/A1348 --mode=link cpCal/instcal/*
root INFO: Loading config overrride file '/Users/jmcclear/Research/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/obs_decam/14.0-4-g3127c16+22/config/ingest.py’
CameraMapper INFO: Loading Posix exposure registry from /Users/jmcclear/Research/A1348
ingest.parse WARN: Unable to find value for ccdnum (derived from CCDNUM)
ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM)
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/instcal/c4d_130204_063637_ooi_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175045/instcal0175045.fits.fz
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/dqmask/c4d_130204_063637_ood_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175045/dqmask0175045.fits.fz
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/wtmap/c4d_130204_063637_oow_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175045/wtmap0175045.fits.fz
ingest.parse WARN: Unable to find value for ccdnum (derived from CCDNUM)
ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM)
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/instcal/c4d_130204_064309_ooi_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175046/instcal0175046.fits.fz
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/dqmask/c4d_130204_064309_ood_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175046/dqmask0175046.fits.fz
ingest INFO: /Users/jmcclear/Research/A1348/cpCal/wtmap/c4d_130204_064309_oow_i_v1.fits.fz ––> /Users/jmcclear/Research/A1348/0175046/wtmap0175046.fits.fz

I didn’t use the config override file, perhaps that’s the problem… EDIT nope, IngestConfig object has no ‘isr’ attribute! nvm.

Anyway, going to the processCcd step,

$processCcd.py /Users/jmcclear/Research/A1348 --id visit=0175045 ccdnum=18 --output /Users/jmcclear/Research/A1348/outDir/ -C override_config.py --config calibrate.doAstrometry=False calibrate.doPhotoCal=False
config overrride file '/Users/jmcclear/Research/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/obs_decam/14.0-4-g3127c16+22/config/processCcd.py’
Loading exposure registry from /Users/jmcclear/Research/A1348/registry.sqlite3
Stripping PVi_j keys from projection RA—TPV/DEC–TPV
/Users/jmcclear/Research/lsst_stack/stack/miniconda3-4.3.21-10a4fa6/DarwinX86/pipe_tasks/14.0-27-g4464a260+2/bin/processCcd.py /Users/jmcclear/Research/A1348 --id visit=0175045 ccdnum=18 --output /Users/jmcclear/Research/A1348/outDir/ -C override_config.py --config calibrate.doAstrometry=False calibrate.doPhotoCal=False
Processing {‘visit’: 175045, ‘ccdnum’: 18, ‘hdu’: 18}
INFO: Loading DECam community pipeline file {‘visit’: 175045, ‘ccdnum’: 18, ‘hdu’: 18}
Stripping PVi_j keys from projection RA—TPV/DEC–TPV
INFO: Processing {‘visit’: 175045, ‘ccdnum’: 18, ‘hdu’: 18}
WARN: Source catalog detected and measured with placeholder or default PSF
INFO: Identified 7 cosmic rays.
INFO: Detected 689 positive sources to 50 sigma.
INFO: Resubtracting the background after object detection
INFO: Measuring 689 sources (689 parents, 0 children)
INFO: Measuring PSF
asurePsf.reserve INFO: Reserved 0/154 sources
INFO: PSF star selector found 154 candidates
INFO: PSF determination using 117/154 stars.
INFO: iter 1; PSF sigma=1.37, dimensions=(25, 25); median background=2865.96
WARN: Source catalog detected and measured with placeholder or default PSF
INFO: Identified 7 cosmic rays.
INFO: Detected 718 positive sources to 50 sigma.
INFO: Resubtracting the background after object detection
INFO: Measuring 718 sources (718 parents, 0 children)
INFO: Measuring PSF
asurePsf.reserve INFO: Reserved 0/154 sources
INFO: PSF star selector found 154 candidates
INFO: PSF determination using 99/154 stars.
INFO: iter 2; PSF sigma=1.37, dimensions=(25, 25); median background=2866.00
INFO: Identified 6 cosmic rays.
INFO: Measuring 718 sources (718 parents, 0 children)
INFO: Measuring aperture corrections for 2 flux fields
INFO: Aperture correction for base_GaussianFlux: RMS 0.025656 from 98
INFO: Aperture correction for base_PsfFlux: RMS 0.023181 from 96
INFO: Applying aperture corrections to 2 flux fields
INFO: Processing {‘visit’: 175045, ‘ccdnum’: 18, ‘hdu’: 18}
INFO: Detected 3788 positive sources to 5 sigma.
INFO: Resubtracting the background after object detection
INFO: Deblending 3788 sources
INFO: Deblended: of 3788 sources, 644 were deblended, creating 2751 children, total 6539 sources
INFO: Measuring 6539 sources (3788 parents, 2751 children)
11, backtrace follows:
0x0000000106a1c5b1 lsst::utils::(anonymous namespace)::signalHandler(int) + 81
0x00007fff7606af5a (null) + 26
0x00007fff7606af5a _sigtramp + 26
lyLookupTable.so" rel="nofollow noopener">applyLookupTable.so 0x000000010328373d pybind11_object_dealloc + 605
0x0000000128ec084e lsst::afw::detection::HeavyFootprint<float, int, float>::~HeavyFootprint() + 46
0x00007fff73e598fd std::__1::__shared_weak_count::__release_shared() + 43
lyLookupTable.so" rel="nofollow noopener">applyLookupTable.so 0x000000010328350b pybind11_object_dealloc + 43
0x000000010089e604 frame_dealloc + 132
0x0000000100942bb8 fast_function + 408
0x0000000100941901 call_function + 401
0x000000010093e0d2 (null) + 27362
0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942f42 (null) + 738
0x0000000100942f42 _PyFunction_FastCallDict + 738
0x0000000100876e87 (null) + 247
0x0000000100876e87 _PyObject_FastCallDict + 247
0x0000000100876fa5 (null) + 149
0x0000000100876fa5 _PyObject_Call_Prepend + 149
0x0000000100876cc0 PyObject_Call + 96
0x000000010093e3b5 (null) + 28101
0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942f42 (null) + 738
0x0000000100942f42 _PyFunction_FastCallDict + 738
0x0000000100876e87 (null) + 247
0x0000000100876e87 _PyObject_FastCallDict + 247
0x0000000100876fa5 (null) + 149
0x0000000100876fa5 _PyObject_Call_Prepend + 149
0x0000000100876cc0 PyObject_Call + 96
0x000000010093e3b5 (null) + 28101
0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942c41 fast_function + 545
0x0000000100941901 call_function + 401
0x000000010093e171 (null) + 27521
0x000000010093e171 _PyEval_EvalFrameDefault + 27521
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942c41 fast_function + 545
0x0000000100941901 call_function + 401
0x000000010093e0d2 (null) + 27362
0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x00000001009375a4 PyEval_EvalCodeEx + 100
0x00000001008a02ad function_call + 381
0x0000000100876cc0 PyObject_Call + 96
0x000000010093e3b5 (null) + 28101
0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942c41 fast_function + 545
0x0000000100941901 call_function + 401
0x000000010093e171 (null) + 27521
0x000000010093e171 _PyEval_EvalFrameDefault + 27521
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942c41 fast_function + 545
0x0000000100941901 call_function + 401
0x000000010093e171 (null) + 27521
0x000000010093e171 _PyEval_EvalFrameDefault + 27521
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x00000001009375a4 PyEval_EvalCodeEx + 100
0x00000001008a02ad function_call + 381
0x0000000100876cc0 PyObject_Call + 96
0x000000010093e3b5 (null) + 28101
0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942c41 fast_function + 545
0x0000000100941901 call_function + 401
0x000000010093e171 (null) + 27521
0x000000010093e171 _PyEval_EvalFrameDefault + 27521
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x00000001009375a4 PyEval_EvalCodeEx + 100
0x00000001008a02ad function_call + 381
0x0000000100876cc0 PyObject_Call + 96
0x000000010093e3b5 (null) + 28101
0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942f42 (null) + 738
0x0000000100942f42 _PyFunction_FastCallDict + 738
0x0000000100876e87 (null) + 247
0x0000000100876e87 _PyObject_FastCallDict + 247
0x0000000100876fa5 (null) + 149
0x0000000100876fa5 _PyObject_Call_Prepend + 149
0x0000000100876cc0 PyObject_Call + 96
0x000000010093e3b5 (null) + 28101
0x000000010093e3b5 _PyEval_EvalFrameDefault + 28101
0x0000000100942dbc (null) + 348
0x0000000100942dbc _PyFunction_FastCallDict + 348
0x0000000100876e87 (null) + 247
0x0000000100876e87 _PyObject_FastCallDict + 247
0x0000000100876fa5 (null) + 149
0x0000000100876fa5 _PyObject_Call_Prepend + 149
0x0000000100876cc0 PyObject_Call + 96
0x00000001008d6e15 slot_tp_call + 117
0x0000000100876e55 (null) + 197
0x0000000100876e55 _PyObject_FastCallDict + 197
0x000000010093323a map_next + 154
0x00000001008a22da listextend + 666
0x00000001008a3723 list_init + 163
0x00000001008d3eb9 type_call + 313
0x0000000100876e55 (null) + 197
0x0000000100876e55 _PyObject_FastCallDict + 197
0x0000000100941828 call_function + 184
0x000000010093e0d2 (null) + 27362
0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
0x0000000100942b9d fast_function + 381
0x0000000100941901 call_function + 401
0x000000010093e0d2 (null) + 27362
0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100942c41 fast_function + 545
0x0000000100941901 call_function + 401
0x000000010093e0d2 (null) + 27362
0x000000010093e0d2 _PyEval_EvalFrameDefault + 27362
0x000000010094238f (null) + 2447
0x000000010094238f _PyEval_EvalCodeWithName + 2447
0x0000000100937534 PyEval_EvalCode + 100
0x000000010096d511 PyRun_FileExFlags + 209
0x000000010096ccc3 PyRun_SimpleFileExFlags + 851
0x0000000100985c49 Py_Main + 3529
0x0000000100866df8 main + 232
0x00007fff75dea145 start + 1
0x000000000000000d 0x0 + 13

hsinfang · December 6, 2017, 9:36pm

Would you obtain the same error if you use the obs_decam of your lsst_distrib installation instead of the local one? In other words, is there any differences if you skip everything in your second code block? (obs_decam is part of lsst_distrib)

rbliu · December 7, 2017, 4:32am

Yes, processCcd.py (calibrate.py as well) should be always located in pipe_tasks instead of obs_*. It is a good point to repeat the error only with setup lsst_distrib (but no setup obs_decam separately) before you run ingestImagesDecam.py and processCcd.py.
I have obs_decam included in a default v13.0 installation. So I guess it should still be there for a v14.0 version.

jmac · December 12, 2017, 3:34am

Hi all,

Thanks for the good suggestions, but unfortunately nothing has worked so far. I was going to try reducing the DECam data from scratch (i.e., ingesting calibration files) to see if that made a difference, but that seems to not be working at the moment. At least, it’s not clear to me what’s missing from the documentation here: https://github.com/lsst/obs_decam. The ingestCalibs can’t find a CALIB_ID on the flat fields I downloaded from portal-nvo.noao.edu, it also doesn’t know what a “fringe file” is, and those

ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM)

warnings still persist. I’m not sure if any of this is related to the segfault above, but could it be? Is there some new processing on the DECam CP end that’s not recognized by existing obs_decam? I know PLVER gets updated periodically…

ctslater · December 12, 2017, 6:27pm

So I think Jim’s comment in the previous thread still applies here; our prior about segfaults is that they most frequently come from some mismatch of various library versions. This backtrace is also very suspicious in that it mentions applyLookupTable.so, which is completely unrelated to measurement (where the crash occurs) and which should never be called given that you’re using community pipeline data (applyLookupTable is for linearity correction). This makes it hard to tell if the backtrace is leading us to the problem or just reflects some sort of corruption issue.

So there are a few options. One is to hunt down this issue further, which I think means increasing the log level so to DEBUG or TRACE so that we can pin down where exactly the segfault happens. It would be good to double check your compiler versions too. I don’t really know if we will be able to get to a good answer that way, but it would at least be a prerequisite. Another option is to delete and reinstall the stack, on the theory that something is either damaged or misinstalled (it sounds like you did all the right steps though). An interesting option might be to try the stack Docker images, since they will completely isolate the environment the stack sees from anything else you have installed. If the docker images still crash, then we have a serious problem!

jmac · December 12, 2017, 8:50pm

Hi all,

here’s the part where I have to cover myself with shame. Clicking around on the Docker page that @ctslater linked led me here: https://pipelines.lsst.io/install/package-development.html. It turns out that without running

setup obs_decam

after setting up the DMstack environment as follows,

source loadLSST.bash
setup lsst_distrib

the "eups list -s " command showed that I was still running the lsst_distrib version of obs_decam, not the updated version available from GitHub. Simply sourcing this accurately fixed the problem! I now have beautiful src.fits files.

I’m really sorry to have bothered everyone with something that now appears so trivial!

Perhaps for the sake of any other offworlders joining the party and to prevent future wastes of time for you, it might be useful to update the documentation both for obs_decam and for lsst.io. Perhaps this is obvious to everyone, but perhaps not. We could state that you (a) have to open a new terminal window, as I found that trying to run obs_decam fresh off the install leads to the same error, and (b) you have to explicitly source any new packages (i.e. obs_decam) after sourcing lsst_distrib. If it would be helpful, I’d be happy to do this myself!

In any case, thank you all so much for your time and help!

jsick · December 12, 2017, 9:05pm

Thanks for your input @jmac. If you’d like to suggest changes, you can open a GitHub Pull Request at https://github.com/lsst/pipelines_lsst_io and I’ll help you merge those in. Otherwise, I can work on that documentation and get feedback from you on whether we’re doing better.

Edit: Looking at the thread, another thing we could improve is showing how to install different cameras, like obs_decam, without having to build it from source. That way you could get the v14 version of obs_decam alongside the v14 version of the rest of the stack. I’ll look into this.

jmac · December 12, 2017, 9:52pm

Hi Jonathan!

I’d love to actually contribute something to the project instead of just being deadweight, so I’ll go ahead with the pull request and try to merge in with the existing documentation (if that’s the right place to do it).

Having a section on how to install different cameras would be useful too, since my group’s medium-term goal is to write up an “obs_superbit” module (https://sites.physics.utoronto.ca/bit) and, in time, make it available as well!

ctslater · December 12, 2017, 11:12pm

Perfect! Glad it worked.