Bad termination from running singleFrameDriver.py via hscPipe v7.5

Charles · June 11, 2019, 1:51pm

Hi,

I ran the singleFrameDriver.py after I use ingestCalibs.py to create calibs as Paul’s suggestion. The process looks OK in the begining, but it was terminated by EXIT CODE: 139. It doesn’t look like a problem caused by calibration setting. Here is the full command I executed:
singleFrameDriver.py $WORKDIR --rerun $RERUNFRAME --calib=$WORKDIR/CALIB/ --id visit=174618 --config processCcd.isr.doBias=False processCcd.isr.doDark=False processCcd.isr.doFringe=False processCcd.isr.doAttachTransmissionCurve=False processCcd.isr.doBrighterFatter=False --job reduceFrames.g.JTL503.hp5 --cores $RCHP5 --clobber-config

This is full log that display on the screen:

root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.2-hsc/config/singleFrameDriver.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.2-hsc/config/hsc/singleFrameDriver.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.2-hsc/config/singleFrameDriver.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.2-hsc/config/hsc/singleFrameDriver.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
root INFO: Running: -c /array/users/ytchen/processed/2019/201904-pi-hp75 --rerun JT-201904-pi-hp75 --calib=/array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/ --id visit=174618 --config processCcd.isr.doBias=False processCcd.isr.doDark=False processCcd.isr.doFringe=False processCcd.isr.doAttachTransmissionCurve=False processCcd.isr.doBrighterFatter=False --clobber-config --noExit
root INFO: Processing 112 targets with a pool of 4 processes…
139623 INFO 2019-06-11T13:38:51.188+0800 CameraMapper: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
139623 INFO 2019-06-11T13:38:51.190+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139622 INFO 2019-06-11T13:38:51.197+0800 CameraMapper: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
139622 INFO 2019-06-11T13:38:51.199+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139621 INFO 2019-06-11T13:38:51.205+0800 CameraMapper: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
139621 INFO 2019-06-11T13:38:51.206+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139623 INFO 2019-06-11T13:38:52.268+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139622 INFO 2019-06-11T13:38:52.299+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139621 INFO 2019-06-11T13:38:52.301+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139622 INFO 2019-06-11T13:38:53.166+0800 singleFrameDriver: ichiro.asiaa.sinica.edu.tw:139622: Start processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 1, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139622 INFO 2019-06-11T13:38:53.166+0800 singleFrameDriver.processCcd: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 1, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139622 INFO 2019-06-11T13:38:53.167+0800 singleFrameDriver.processCcd.isr: Performing ISR on sensor {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 1, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 INFO 2019-06-11T13:38:53.174+0800 singleFrameDriver: ichiro.asiaa.sinica.edu.tw:139621: Start processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 INFO 2019-06-11T13:38:53.175+0800 singleFrameDriver.processCcd: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 INFO 2019-06-11T13:38:53.175+0800 singleFrameDriver.processCcd.isr: Performing ISR on sensor {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139623 INFO 2019-06-11T13:38:53.206+0800 singleFrameDriver: ichiro.asiaa.sinica.edu.tw:139623: Start processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 2, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139623 INFO 2019-06-11T13:38:53.206+0800 singleFrameDriver.processCcd: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 2, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139623 INFO 2019-06-11T13:38:53.206+0800 singleFrameDriver.processCcd.isr: Performing ISR on sensor {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 2, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 INFO 2019-06-11T13:38:55.278+0800 singleFrameDriver.processCcd.isr: Converting exposure to floating point values
139621 WARN 2019-06-11T13:38:55.450+0800 singleFrameDriver.processCcd.isr: Amplifier 1 is bad.
139622 INFO 2019-06-11T13:38:55.461+0800 singleFrameDriver.processCcd.isr: Converting exposure to floating point values
139623 INFO 2019-06-11T13:38:55.495+0800 singleFrameDriver.processCcd.isr: Converting exposure to floating point values
139621 INFO 2019-06-11T13:38:55.552+0800 singleFrameDriver.processCcd.isr: Assembling CCD from amplifiers
139622 INFO 2019-06-11T13:38:55.733+0800 singleFrameDriver.processCcd.isr: Assembling CCD from amplifiers
139623 INFO 2019-06-11T13:38:55.783+0800 singleFrameDriver.processCcd.isr: Assembling CCD from amplifiers
139621 INFO 2019-06-11T13:38:56.102+0800 singleFrameDriver.processCcd.isr: Applying linearizer.
139621 INFO 2019-06-11T13:38:56.138+0800 singleFrameDriver.processCcd.isr: Applying crosstalk correction.
139621 INFO 2019-06-11T13:38:56.138+0800 singleFrameDriver.processCcd.isr.crosstalk: Applying crosstalk correction/Subaru
139622 INFO 2019-06-11T13:38:56.228+0800 singleFrameDriver.processCcd.isr: Applying linearizer.
139622 INFO 2019-06-11T13:38:56.255+0800 singleFrameDriver.processCcd.isr: Applying crosstalk correction.
139622 INFO 2019-06-11T13:38:56.255+0800 singleFrameDriver.processCcd.isr.crosstalk: Applying crosstalk correction/Subaru
139623 INFO 2019-06-11T13:38:56.307+0800 singleFrameDriver.processCcd.isr: Applying linearizer.
139623 INFO 2019-06-11T13:38:56.336+0800 singleFrameDriver.processCcd.isr: Applying crosstalk correction.
139623 INFO 2019-06-11T13:38:56.336+0800 singleFrameDriver.processCcd.isr.crosstalk: Applying crosstalk correction/Subaru
139621 INFO 2019-06-11T13:38:57.049+0800 singleFrameDriver.processCcd.isr: Widening saturation trails.
139621 INFO 2019-06-11T13:38:57.065+0800 singleFrameDriver.processCcd.isr: Applying flat correction.
139621 INFO 2019-06-11T13:38:57.121+0800 singleFrameDriver.processCcd.isr: Masking and interpolating defects.
139622 INFO 2019-06-11T13:38:57.133+0800 singleFrameDriver.processCcd.isr: Widening saturation trails.
139622 INFO 2019-06-11T13:38:57.159+0800 singleFrameDriver.processCcd.isr: Applying flat correction.
139623 INFO 2019-06-11T13:38:57.201+0800 singleFrameDriver.processCcd.isr: Widening saturation trails.
139622 INFO 2019-06-11T13:38:57.214+0800 singleFrameDriver.processCcd.isr: Masking and interpolating defects.
139623 INFO 2019-06-11T13:38:57.215+0800 singleFrameDriver.processCcd.isr: Applying flat correction.
139623 INFO 2019-06-11T13:38:57.264+0800 singleFrameDriver.processCcd.isr: Masking and interpolating defects.
139621 INFO 2019-06-11T13:38:57.516+0800 singleFrameDriver.processCcd.isr: Interpolating saturated pixels.
139622 INFO 2019-06-11T13:38:57.676+0800 singleFrameDriver.processCcd.isr: Interpolating saturated pixels.
139623 INFO 2019-06-11T13:38:57.687+0800 singleFrameDriver.processCcd.isr: Interpolating saturated pixels.
139621 INFO 2019-06-11T13:38:57.968+0800 singleFrameDriver.processCcd.isr: Masking and interpolating NAN value pixels.
139623 INFO 2019-06-11T13:38:58.136+0800 singleFrameDriver.processCcd.isr: Masking and interpolating NAN value pixels.
139622 INFO 2019-06-11T13:38:58.144+0800 singleFrameDriver.processCcd.isr: Masking and interpolating NAN value pixels.
139621 INFO 2019-06-11T13:38:58.256+0800 singleFrameDriver.processCcd.isr: Setting rough magnitude zero point: 32.692803
139621 INFO 2019-06-11T13:38:58.256+0800 singleFrameDriver.processCcd.isr: Constructing Vignette polygon.
139621 INFO 2019-06-11T13:38:58.279+0800 singleFrameDriver.processCcd.isr: Adding a distortion model to the WCS.
139621 INFO 2019-06-11T13:38:58.281+0800 singleFrameDriver.processCcd.isr: Measuring background level:
139623 INFO 2019-06-11T13:38:58.460+0800 singleFrameDriver.processCcd.isr: Setting rough magnitude zero point: 32.692803
139623 INFO 2019-06-11T13:38:58.460+0800 singleFrameDriver.processCcd.isr: Constructing Vignette polygon.
139623 INFO 2019-06-11T13:38:58.461+0800 singleFrameDriver.processCcd.isr: Adding a distortion model to the WCS.
139623 INFO 2019-06-11T13:38:58.464+0800 singleFrameDriver.processCcd.isr: Measuring background level:
139622 INFO 2019-06-11T13:38:58.472+0800 singleFrameDriver.processCcd.isr: Setting rough magnitude zero point: 32.692803
139622 INFO 2019-06-11T13:38:58.472+0800 singleFrameDriver.processCcd.isr: Constructing Vignette polygon.
139622 INFO 2019-06-11T13:38:58.473+0800 singleFrameDriver.processCcd.isr: Adding a distortion model to the WCS.
139622 INFO 2019-06-11T13:38:58.476+0800 singleFrameDriver.processCcd.isr: Measuring background level:
139621 INFO 2019-06-11T13:38:58.585+0800 singleFrameDriver.processCcd.isr: Flattened sky level: 305.868256 +/- 10.445378
139623 INFO 2019-06-11T13:38:58.808+0800 singleFrameDriver.processCcd.isr: Flattened sky level: 304.669098 +/- 10.142621
139622 INFO 2019-06-11T13:38:58.831+0800 singleFrameDriver.processCcd.isr: Flattened sky level: 306.349213 +/- 9.998964
139621 INFO 2019-06-11T13:38:58.852+0800 singleFrameDriver.processCcd.isr: Measuring sky levels in 8x16 grids: 306.108966
139621 INFO 2019-06-11T13:38:58.852+0800 singleFrameDriver.processCcd.isr: Sky flatness in 8x16 grids - pp: 0.017215 rms: 0.003908
139623 INFO 2019-06-11T13:38:59.146+0800 singleFrameDriver.processCcd.isr: Measuring sky levels in 8x16 grids: 304.970223
139623 INFO 2019-06-11T13:38:59.146+0800 singleFrameDriver.processCcd.isr: Sky flatness in 8x16 grids - pp: 0.034628 rms: 0.004833
139622 INFO 2019-06-11T13:38:59.172+0800 singleFrameDriver.processCcd.isr: Measuring sky levels in 8x16 grids: 306.399288
139622 INFO 2019-06-11T13:38:59.172+0800 singleFrameDriver.processCcd.isr: Sky flatness in 8x16 grids - pp: 0.023681 rms: 0.004674
139621 INFO 2019-06-11T13:38:59.186+0800 singleFrameDriver.processCcd.charImage: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139623 INFO 2019-06-11T13:38:59.498+0800 singleFrameDriver.processCcd.charImage: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 2, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139622 INFO 2019-06-11T13:38:59.525+0800 singleFrameDriver.processCcd.charImage: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 1, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 WARN 2019-06-11T13:38:59.694+0800 singleFrameDriver.processCcd.charImage: Source catalog detected and measured with placeholder or default PSF
139621 INFO 2019-06-11T13:39:00.075+0800 singleFrameDriver.processCcd.charImage.repair: Identified 42 cosmic rays.
139623 WARN 2019-06-11T13:39:00.090+0800 singleFrameDriver.processCcd.charImage: Source catalog detected and measured with placeholder or default PSF
139622 WARN 2019-06-11T13:39:00.135+0800 singleFrameDriver.processCcd.charImage: Source catalog detected and measured with placeholder or default PSF
139623 INFO 2019-06-11T13:39:00.478+0800 singleFrameDriver.processCcd.charImage.repair: Identified 69 cosmic rays.
139622 INFO 2019-06-11T13:39:00.496+0800 singleFrameDriver.processCcd.charImage.repair: Identified 78 cosmic rays.
139621 INFO 2019-06-11T13:39:01.319+0800 singleFrameDriver.processCcd.charImage.detection: Detected 129 positive peaks in 35 footprints and 0 negative peaks in 0 footprints to 50 sigma
139621 INFO 2019-06-11T13:39:01.565+0800 singleFrameDriver.processCcd.charImage.detection: Resubtracting the background after object detection
139622 INFO 2019-06-11T13:39:01.799+0800 singleFrameDriver.processCcd.charImage.detection: Detected 341 positive peaks in 54 footprints and 0 negative peaks in 0 footprints to 50 sigma
139623 INFO 2019-06-11T13:39:01.855+0800 singleFrameDriver.processCcd.charImage.detection: Detected 256 positive peaks in 58 footprints and 0 negative peaks in 0 footprints to 50 sigma
139622 INFO 2019-06-11T13:39:02.133+0800 singleFrameDriver.processCcd.charImage.detection: Resubtracting the background after object detection
139623 INFO 2019-06-11T13:39:02.190+0800 singleFrameDriver.processCcd.charImage.detection: Resubtracting the background after object detection
139621 INFO 2019-06-11T13:39:02.348+0800 singleFrameDriver.processCcd.charImage.measurement: Measuring 35 sources (35 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7f843fbaabd4]
/lib64/libc.so.6(+0x35990) [0x7f845611d990]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.2-hsc/lib/libafw.so(lsst::afw::table::AliasMap::_apply(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) const+0x78) [0x7f84336a7958]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.2-hsc/lib/libafw.so(lsst::afw::table::SchemaItemlsst::afw::table::Flag lsst::afw::table::Schema::findlsst::afw::table::Flag(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const+0x3c) [0x7f84336f986c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::meas::base::SdssCentroidAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0xe3) [0x7f8409a5e803]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1f9aed) [0x7f83e9017aed]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1fb897) [0x7f83e9019897]
python((null)+0x264) [0x557bb7abde24]
python(_PyMethodDef_RawFastCallKeywords+0x264) [0x557bb7abde24]
python((null)+0x21) [0x557bb7abdf41]
python(_PyCFunction_FastCallKeywords+0x21) [0x557bb7abdf41]
python((null)+0x5266) [0x557bb7b18d26]
python(_PyEval_EvalFrameDefault+0x5266) [0x557bb7b18d26]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x325) [0x557bb7abd0f5]
python(_PyFunction_FastCallKeywords+0x325) [0x557bb7abd0f5]
python((null)+0x530) [0x557bb7b13ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x557bb7b13ff0]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x3dc) [0x557bb7a5a3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x557bb7a5a3bc]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x3dc) [0x557bb7a5a3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x557bb7a5a3bc]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x3dc) [0x557bb7a5a3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x557bb7a5a3bc]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x3dc) [0x557bb7a5a3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x557bb7a5a3bc]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x530) [0x557bb7b13ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x557bb7b13ff0]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0xfb) [0x557bb7abcecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x557bb7abcecb]
python((null)+0x530) [0x557bb7b13ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x557bb7b13ff0]
python((null)+0xfb) [0x557bb7abcecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x557bb7abcecb]
python((null)+0x4aa1) [0x557bb7b18561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x557bb7b18561]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(+0x17194a) [0x557bb7ab594a]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0xab9) [0x557bb7a59899]
python(_PyEval_EvalCodeWithName+0xab9) [0x557bb7a59899]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x6e6) [0x557bb7b141a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x557bb7b141a6]
python((null)+0xab9) [0x557bb7a59899]
python(_PyEval_EvalCodeWithName+0xab9) [0x557bb7a59899]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x6e6) [0x557bb7b141a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x557bb7b141a6]
python((null)+0xab9) [0x557bb7a59899]
python(_PyEval_EvalCodeWithName+0xab9) [0x557bb7a59899]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x530) [0x557bb7b13ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x557bb7b13ff0]
python(+0x178d3b) [0x557bb7abcd3b]
python((null)+0x6e6) [0x557bb7b141a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x557bb7b141a6]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x4aa1) [0x557bb7b18561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x557bb7b18561]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python(PyEval_EvalCodeEx+0x44) [0x557bb7a59fa4]
python(PyEval_EvalCode+0x1c) [0x557bb7a59fcc]
python(+0x22f664) [0x557bb7b73664]
python(PyRun_StringFlags+0x7d) [0x557bb7b7e77d]
python(PyRun_SimpleStringFlags+0x3f) [0x557bb7b7e7df]
python(+0x23a8dd) [0x557bb7b7e8dd]
python((null)+0x3c) [0x557bb7b7ec4c]
python(_Py_UnixMain+0x3c) [0x557bb7b7ec4c]
/lib64/libc.so.6((null)+0xf1) [0x7f8456108401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7f8456108401]
python(+0x1df982) [0x557bb7b23982]
139622 INFO 2019-06-11T13:39:02.664+0800 singleFrameDriver.processCcd.charImage.measurement: Measuring 54 sources (54 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7f296f986bd4]
/lib64/libc.so.6(+0x35990) [0x7f2985ef9990]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::afw::table::SourceRecord::getCentroidFlag() const+0x5d) [0x7f293982078d]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::meas::base::SafeCentroidExtractor::operator()(lsst::afw::table::SourceRecord&, lsst::meas::base::FlagHandler const&) const+0x2a3) [0x7f293981ff53]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::meas::base::SdssCentroidAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0x48) [0x7f293983a768]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1f9aed) [0x7f2918df3aed]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1fb897) [0x7f2918df5897]
python((null)+0x264) [0x55b73a541e24]
python(_PyMethodDef_RawFastCallKeywords+0x264) [0x55b73a541e24]
python((null)+0x21) [0x55b73a541f41]
python(_PyCFunction_FastCallKeywords+0x21) [0x55b73a541f41]
python((null)+0x5266) [0x55b73a59cd26]
python(_PyEval_EvalFrameDefault+0x5266) [0x55b73a59cd26]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x325) [0x55b73a5410f5]
python(_PyFunction_FastCallKeywords+0x325) [0x55b73a5410f5]
python((null)+0x530) [0x55b73a597ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b73a597ff0]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x3dc) [0x55b73a4de3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b73a4de3bc]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x3dc) [0x55b73a4de3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b73a4de3bc]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x3dc) [0x55b73a4de3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b73a4de3bc]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x3dc) [0x55b73a4de3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b73a4de3bc]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x530) [0x55b73a597ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b73a597ff0]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0xfb) [0x55b73a540ecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55b73a540ecb]
python((null)+0x530) [0x55b73a597ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b73a597ff0]
python((null)+0xfb) [0x55b73a540ecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55b73a540ecb]
python((null)+0x4aa1) [0x55b73a59c561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55b73a59c561]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(+0x17194a) [0x55b73a53994a]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0xab9) [0x55b73a4dd899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b73a4dd899]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x6e6) [0x55b73a5981a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b73a5981a6]
python((null)+0xab9) [0x55b73a4dd899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b73a4dd899]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x6e6) [0x55b73a5981a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b73a5981a6]
python((null)+0xab9) [0x55b73a4dd899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b73a4dd899]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x530) [0x55b73a597ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b73a597ff0]
python(+0x178d3b) [0x55b73a540d3b]
python((null)+0x6e6) [0x55b73a5981a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b73a5981a6]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x4aa1) [0x55b73a59c561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55b73a59c561]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python(PyEval_EvalCodeEx+0x44) [0x55b73a4ddfa4]
python(PyEval_EvalCode+0x1c) [0x55b73a4ddfcc]
python(+0x22f664) [0x55b73a5f7664]
python(PyRun_StringFlags+0x7d) [0x55b73a60277d]
python(PyRun_SimpleStringFlags+0x3f) [0x55b73a6027df]
python(+0x23a8dd) [0x55b73a6028dd]
python((null)+0x3c) [0x55b73a602c4c]
python(_Py_UnixMain+0x3c) [0x55b73a602c4c]
/lib64/libc.so.6((null)+0xf1) [0x7f2985ee4401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7f2985ee4401]
python(+0x1df982) [0x55b73a5a7982]
139623 INFO 2019-06-11T13:39:02.697+0800 singleFrameDriver.processCcd.charImage.measurement: Measuring 58 sources (58 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7f17e8ccbbd4]
/lib64/libc.so.6(+0x35990) [0x7f17ff23e990]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.2-hsc/lib/libafw.so(lsst::afw::table::AliasMap::_apply(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) const+0x78) [0x7f17dc7c8958]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.2-hsc/lib/libafw.so(lsst::afw::table::SchemaItemlsst::afw::table::Flag lsst::afw::table::Schema::findlsst::afw::table::Flag(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const+0x3c) [0x7f17dc81a86c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::meas::base::SdssCentroidAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0xe3) [0x7f17b2b7f803]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1f9aed) [0x7f1792138aed]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1fb897) [0x7f179213a897]
python((null)+0x264) [0x55b06c52ee24]
python(_PyMethodDef_RawFastCallKeywords+0x264) [0x55b06c52ee24]
python((null)+0x21) [0x55b06c52ef41]
python(_PyCFunction_FastCallKeywords+0x21) [0x55b06c52ef41]
python((null)+0x5266) [0x55b06c589d26]
python(_PyEval_EvalFrameDefault+0x5266) [0x55b06c589d26]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x325) [0x55b06c52e0f5]
python(_PyFunction_FastCallKeywords+0x325) [0x55b06c52e0f5]
python((null)+0x530) [0x55b06c584ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b06c584ff0]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x3dc) [0x55b06c4cb3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b06c4cb3bc]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x3dc) [0x55b06c4cb3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b06c4cb3bc]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x3dc) [0x55b06c4cb3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b06c4cb3bc]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x3dc) [0x55b06c4cb3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b06c4cb3bc]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x530) [0x55b06c584ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b06c584ff0]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0xfb) [0x55b06c52decb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55b06c52decb]
python((null)+0x530) [0x55b06c584ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b06c584ff0]
python((null)+0xfb) [0x55b06c52decb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55b06c52decb]
python((null)+0x4aa1) [0x55b06c589561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55b06c589561]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(+0x17194a) [0x55b06c52694a]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0xab9) [0x55b06c4ca899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b06c4ca899]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x6e6) [0x55b06c5851a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b06c5851a6]
python((null)+0xab9) [0x55b06c4ca899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b06c4ca899]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x6e6) [0x55b06c5851a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b06c5851a6]
python((null)+0xab9) [0x55b06c4ca899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b06c4ca899]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x530) [0x55b06c584ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b06c584ff0]
python(+0x178d3b) [0x55b06c52dd3b]
python((null)+0x6e6) [0x55b06c5851a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b06c5851a6]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x4aa1) [0x55b06c589561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55b06c589561]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python(PyEval_EvalCodeEx+0x44) [0x55b06c4cafa4]
python(PyEval_EvalCode+0x1c) [0x55b06c4cafcc]
python(+0x22f664) [0x55b06c5e4664]
python(PyRun_StringFlags+0x7d) [0x55b06c5ef77d]
python(PyRun_SimpleStringFlags+0x3f) [0x55b06c5ef7df]
python(+0x23a8dd) [0x55b06c5ef8dd]
python((null)+0x3c) [0x55b06c5efc4c]
python(_Py_UnixMain+0x3c) [0x55b06c5efc4c]
/lib64/libc.so.6((null)+0xf1) [0x7f17ff229401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7f17ff229401]
python(+0x1df982) [0x55b06c594982]
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 139622 RUNNING AT ichiro.asiaa.sinica.edu.tw
= EXIT CODE: 139
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

ktl · June 11, 2019, 11:29pm

That’s a very odd place to get a segmentation fault, as it’s doing standard C++ library manipulation of standard C++ strings (using a standard C++ map). Segfaults often occur because of a mismatch in versions between pieces of code. Are you sure that the libc you are using is the one the stack was built with? Are there other potential incompatibilities?

Charles · June 12, 2019, 5:12am

Thanks, that’s is a good point to shooting this problem. I do remember that our administrator encountered some incompatible issues during installation. I will check the library with the administrator first.

Charles · June 18, 2019, 6:53am

Our administrator reply that he followed all the instruction of the installation and have no compatibility error/issue during the installation process. I had tried v7.4, but got the same termination. I only see one version libc in the same directory (/lib64/), which was linked to libc-2.24.so. I am really confused by this issue…

Charles · June 20, 2019, 1:57am

Is there a possible solution for this issue? Or is there anyone installed >v7.5, then successfully processed a frame with singleFrameDriver.py?

price · June 20, 2019, 2:59pm

Sorry for the late response; I didn’t see this thread.

We have processed a lot of data with hscPipe 7 for the HSC SSP, so I’m inclined to agree with @ktl that the installation and setup is the first place to look. Perhaps you could post exactly how the pipeline was installed? Most importantly, are you using the new binary install?

Do you get the same problem when you use processCcd.py instead of singleFrameDriver.py?

Charles · June 21, 2019, 2:10am

I had wrote an email to ask administrator about the exactly steps of installation.

In the meantime, I tested processCcd.py with following parameters, then got the same problem:
processCcd.py $WORKDIR --rerun $RERUNFRAME --calib=$WORKDIR/CALIB/ --id visit=175292 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False --clobber-config

Output message:

root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/processCcd.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/hsc/processCcd.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp772/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/calibRegistry.sqlite3
root INFO: Running: /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.0-hsc+1/bin/processCcd.py /array/users/ytchen/processed/2019/201904-pi-hp772 --rerun JT-201904-pi-hp772 --calib=/array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/ --id visit=175292 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False --clobber-config
processCcd INFO: Processing {‘visit’: 175292, ‘field’: ‘JTL505’, ‘dateObs’: ‘2019-04-11’, ‘pointing’: 2657, ‘filter’: ‘HSC-I2’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-11’, ‘expTime’: 24.22}
processCcd.isr INFO: Performing ISR on sensor {‘visit’: 175292, ‘field’: ‘JTL505’, ‘dateObs’: ‘2019-04-11’, ‘pointing’: 2657, ‘filter’: ‘HSC-I2’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-11’, ‘expTime’: 24.22}
processCcd.isr INFO: Converting exposure to floating point values
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7fdb32d34bd4]
/lib64/libc.so.6(+0x35990) [0x7fdb4093a990]
/lib64/libc.so.6(cfree+0x22) [0x7fdb4098b292]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(+0x76851c) [0x7fdb360f451c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(std::_Sp_counted_ptr_inplace<lsst::afw::detection::PeakRecord, std::allocatorlsst::afw::detection::PeakRecord, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x56) [0x7fdb35c496f6]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/7.0-hsc+1/lib/libmeas_algorithms.so(std::_Sp_counted_ptr_inplace<lsst::afw::detection::Footprint, std::allocatorlsst::afw::detection::Footprint, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x73) [0x7fdb368f9473]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/detection/footprint.so(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x46) [0x7fdaedc8a4d6]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/detection/footprint.so(+0x1f1640) [0x7fdaedc8a640]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc+1/python/lsst/ip/isr/applyLookupTable.so(+0x1f5208) [0x7fdb36fb7208]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc+1/python/lsst/ip/isr/applyLookupTable.so(+0x1f5a6e) [0x7fdb36fb7a6e]
python(+0x10ed42) [0x5640d5ea9d42]
python((null)+0x48b) [0x5640d5eb10ab]
python(_PyEval_EvalCodeWithName+0x48b) [0x5640d5eb10ab]
python((null)+0x387) [0x5640d5f13337]
python(_PyFunction_FastCallKeywords+0x387) [0x5640d5f13337]
python((null)+0x14f5) [0x5640d5f6c5d5]
python(_PyEval_EvalFrameDefault+0x14f5) [0x5640d5f6c5d5]
python((null)+0xba9) [0x5640d5eb17c9]
python(_PyEval_EvalCodeWithName+0xba9) [0x5640d5eb17c9]
python((null)+0x387) [0x5640d5f13337]
python(_PyFunction_FastCallKeywords+0x387) [0x5640d5f13337]
python((null)+0x535) [0x5640d5f6b615]
python(_PyEval_EvalFrameDefault+0x535) [0x5640d5f6b615]
python((null)+0x2f9) [0x5640d5eb0f19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x5640d5eb0f19]
python((null)+0x3d8) [0x5640d5eb21e8]
python(_PyFunction_FastCallDict+0x3d8) [0x5640d5eb21e8]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0x5da) [0x5640d5eb11fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x5640d5eb11fa]
python((null)+0x3d8) [0x5640d5eb21e8]
python(_PyFunction_FastCallDict+0x3d8) [0x5640d5eb21e8]
python((null)+0x63) [0x5640d5ec8cb3]
python(_PyObject_Call_Prepend+0x63) [0x5640d5ec8cb3]
python(PyObject_Call+0x6e) [0x5640d5ebd7de]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0x10b) [0x5640d5eb1f1b]
python(_PyFunction_FastCallDict+0x10b) [0x5640d5eb1f1b]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0x5da) [0x5640d5eb11fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x5640d5eb11fa]
python((null)+0x387) [0x5640d5f13337]
python(_PyFunction_FastCallKeywords+0x387) [0x5640d5f13337]
python((null)+0x535) [0x5640d5f6b615]
python(_PyEval_EvalFrameDefault+0x535) [0x5640d5f6b615]
python((null)+0x10b) [0x5640d5eb1f1b]
python(_PyFunction_FastCallDict+0x10b) [0x5640d5eb1f1b]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0x5da) [0x5640d5eb11fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x5640d5eb11fa]
python((null)+0x1d4) [0x5640d5eb1fe4]
python(_PyFunction_FastCallDict+0x1d4) [0x5640d5eb1fe4]
python((null)+0x63) [0x5640d5ec8cb3]
python(_PyObject_Call_Prepend+0x63) [0x5640d5ec8cb3]
python(PyObject_Call+0x6e) [0x5640d5ebd7de]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0xfb) [0x5640d5f130ab]
python(_PyFunction_FastCallKeywords+0xfb) [0x5640d5f130ab]
python((null)+0x535) [0x5640d5f6b615]
python(_PyEval_EvalFrameDefault+0x535) [0x5640d5f6b615]
python((null)+0x10b) [0x5640d5eb1f1b]
python(_PyFunction_FastCallDict+0x10b) [0x5640d5eb1f1b]
python((null)+0x63) [0x5640d5ec8cb3]
python(_PyObject_Call_Prepend+0x63) [0x5640d5ec8cb3]
python(+0x170cca) [0x5640d5f0bcca]
python((null)+0x93) [0x5640d5eb2403]
python(_PyObject_FastCallDict+0x93) [0x5640d5eb2403]
python(+0x14e942) [0x5640d5ee9942]
python(+0x1bbee4) [0x5640d5f56ee4]
python((null)+0x128) [0x5640d5f142d8]
python(_PyObject_FastCallKeywords+0x128) [0x5640d5f142d8]
python((null)+0x4cb6) [0x5640d5f6fd96]
python(_PyEval_EvalFrameDefault+0x4cb6) [0x5640d5f6fd96]
python((null)+0xfb) [0x5640d5f130ab]
python(_PyFunction_FastCallKeywords+0xfb) [0x5640d5f130ab]
python((null)+0x535) [0x5640d5f6b615]
python(_PyEval_EvalFrameDefault+0x535) [0x5640d5f6b615]
python((null)+0x2f9) [0x5640d5eb0f19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x5640d5eb0f19]
python((null)+0x325) [0x5640d5f132d5]
python(_PyFunction_FastCallKeywords+0x325) [0x5640d5f132d5]
python((null)+0x4b64) [0x5640d5f6fc44]
python(_PyEval_EvalFrameDefault+0x4b64) [0x5640d5f6fc44]
python((null)+0x2f9) [0x5640d5eb0f19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x5640d5eb0f19]
python(PyEval_EvalCodeEx+0x44) [0x5640d5eb1dd4]
python(PyEval_EvalCode+0x1c) [0x5640d5eb1dfc]
python(+0x22f9e4) [0x5640d5fca9e4]
python(PyRun_FileExFlags+0xa1) [0x5640d5fd4bd1]
python(PyRun_SimpleFileExFlags+0x1c3) [0x5640d5fd4dc3]
python(+0x23aedb) [0x5640d5fd5edb]
python((null)+0x3c) [0x5640d5fd5fbc]
python(_Py_UnixMain+0x3c) [0x5640d5fd5fbc]
/lib64/libc.so.6((null)+0xf1) [0x7fdb40925401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7fdb40925401]
python(+0x1dfed2) [0x5640d5f7aed2]
Segmentation fault (core dumped)

ktl · June 21, 2019, 10:16am

That’s in a different part of the code (and a different version of afw: 7.6 vs. 7.2). Note that the first two lines in the backtrace (libutils.so and libc.so.6(+0x35990)) are the common code that generates the backtrace, so it’s really the third line that is where the problem is occurring. Seems to me something is definitely not right with your installation.

ktl · June 21, 2019, 10:22am

And to highlight something @price said, posting the setup command(s) you used or the result of eups list --setup might also help (or at least rule out one source of potential problems).

Charles · June 22, 2019, 3:32pm

OK. Our administrator replied that he cannot use the binary install since the OS of the server is Fedora Core. So he installed the hscPipe using eups, instead of binary:

source loadLSST.bash
eups distrib install hscPipe 7.7.2

The following list is the output of eups list --setup:

afw 7.6-hsc	setup
apr 1.5.2	current setup
apr_util 1.5.4-1-g78a90a1	current setup
astro_metadata_translator 7.1-hsc	setup
astrometry_net 0.67.123ff3e.lsst1+13	current setup
astrometry_net_data 10.0+138	setup
astropy 2.0.1-2-gb0b7760	current setup
astshim 7.3-hsc	setup
base 7.0-hsc	setup
boost 1.60.lsst1-3-g1c4857aa	current setup
cfitsio 3360.lsst5-1-gfc68e3c	current setup
coadd_utils 7.0-hsc+1	setup
coord 1.1.lsst+3	current setup
ctrl_pool 7.0-hsc+1	setup
daf_base 7.0-hsc	current setup
daf_butler 7.0-hsc+1	setup
daf_persistence 7.0-hsc	current setup
dax_ppdb 7.0-hsc+1	setup
display_ds9 7.0-hsc+1	setup
doxygen 1.8.13.lsst1-2-g524ee4e	current setup
eigen 7.2-hsc	setup
esutil 0.6.2.5.lsst1-1-gdae48e7	current setup
fftw 3.3.4.lsst2	current setup
flake8 3.5.0-3-g1298ec5	current setup
galsim 1.5.1.lsst1-11-gfba72ad	current setup
geom 7.0-hsc	current setup
gsl 2.4	current setup
hscPipe 7.7.2	setup
ip_diffim 7.0-hsc+1	setup
ip_isr 7.1-hsc+1	setup
jointcal 7.5-hsc+1	setup
jointcal_cholmod 7.2-hsc	setup
libyaml 0.1.7	current setup
lmfit 0.9.3+12	current setup
log 7.0-hsc	current setup
log4cxx 0.10.0.lsst7+2	current setup
matplotlib 2.0.2-2-gce61d8e	current setup
meas_algorithms 7.0-hsc+1	setup
meas_astrom 7.0-hsc+1	setup
meas_base 7.0-hsc+1	setup
meas_deblender 7.0-hsc+1	setup
meas_extensions_convolved 7.0-hsc+1	setup
meas_extensions_photometryKron 7.0-hsc+1	setup
meas_extensions_psfex 7.0-hsc+1	setup
meas_extensions_shapeHSM 7.0-hsc+1	setup
meas_modelfit 7.0-hsc+1	setup
meas_mosaic 7.0-hsc+4	setup
minuit2 5.34.14	current setup
mpi 0.0.1+3	current setup
mpi4py 2.0.0-1-g6d58a44	current setup
mpich 3.2.1	current setup
ndarray 7.0-hsc-1-g8273558	setup
numpy 1.13.1-2-g36a8a36	current setup
obs_base 7.0-hsc+1	setup
obs_subaru 7.7.2-hsc	setup
obs_test 7.0-hsc+1	setup
pep8_naming 0.4.1-2-ge968bca	current setup
pex_config 7.0-hsc	current setup
pex_exceptions 7.0-hsc	current setup
pex_policy 7.0-hsc	current setup
pipe_base 7.0-hsc+1	setup
pipe_drivers 7.5-hsc+1	setup
pipe_tasks 7.0-hsc+1	setup
proxmin 7.0-hsc	current setup
psfex 7.0-hsc	current setup
pybind11 2.1.1-4-gd10a463	current setup
pycodestyle 2.3.1-6-g0d131b0	current setup
pyflakes 1.6.0-2-gb8a7d8b	current setup
pytest 3.2.0.lsst2-5-ge2a44db	current setup
pytest_cov 2.5.1+7	current setup
pytest_flake8 0.9.1-3-g14151fe	current setup
pytest_forked 0.2.lsst2-3-gc5fda7b	current setup
pytest_session2file 0.1.9+16	current setup
pytest_xdist 1.20.1.lsst2-3-ga186ae7	current setup
python 0.0.7-3-g85c3855	current setup
python_coverage 4.5.1+3	current setup
python_execnet 1.4.1.lsst2-3-gc0d69c6	current setup
python_future 0.16.0+6	current setup
python_mccabe 0.6.1+16	current setup
python_psutil 5.4.3+5	current setup
python_py 1.7.0+3	current setup
pyyaml 3.11.lsst2-2-ga7699fc	current setup
requests 2.9.1.lsst1+7	current setup
scarlet 7.0-hsc	current setup
scipy 0.0.1.lsst1-2-g5722128	current setup
scons 3.0.0.lsst1+6	current setup
sconsUtils 7.0-hsc	current setup
shapelet 7.0-hsc+1	setup
skymap 7.0-hsc+1	setup
sphgeom 7.0-hsc	current setup
sqlalchemy 1.2.16+3	current setup
starlink_ast 7.0-hsc	current setup
utils 7.0-hsc	current setup
verify 17.0.1-1-g84a5949+25	setup
verify_metrics 7.1-hsc	setup
wcslib 5.13.lsst1+4	current setup
xpa 2.1.15.lsst3	current setup

price · June 24, 2019, 2:10am

It seems strange to me that it could be in applyLookupTable.so, when the HSC linearity correction doesn’t use a lookup table. I will investigate further tomorrow, but in the mean time perhaps you could try disabling the linearity correction (isr.doLinearize=False)?

Charles · June 24, 2019, 4:23am

I tried the following command, then got the message of Segmentation fault again in output:

(lsst-scipipe-1172c30) [hscpipe5@ichiro 201904-pi-hp772]$ processCcd.py $WORKDIR --rerun $RERUNFRAME --calib=$WORKDIR/CALIB/ --id visit=175292 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False isr.doLinearize=False --clobber-config
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/processCcd.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/hsc/processCcd.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp772/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/calibRegistry.sqlite3
root INFO: Running: /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.0-hsc+1/bin/processCcd.py /array/users/ytchen/processed/2019/201904-pi-hp772 --rerun JT-201904-pi-hp772 --calib=/array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/ --id visit=175292 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False isr.doLinearize=False --clobber-config
processCcd INFO: Processing {‘visit’: 175292, ‘field’: ‘JTL505’, ‘dateObs’: ‘2019-04-11’, ‘pointing’: 2657, ‘filter’: ‘HSC-I2’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-11’, ‘expTime’: 24.22}
processCcd.isr INFO: Performing ISR on sensor {‘visit’: 175292, ‘field’: ‘JTL505’, ‘dateObs’: ‘2019-04-11’, ‘pointing’: 2657, ‘filter’: ‘HSC-I2’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-11’, ‘expTime’: 24.22}
processCcd.isr INFO: Converting exposure to floating point values
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7fb1c8654bd4]
/lib64/libc.so.6(+0x35990) [0x7fb1d625a990]
/lib64/libc.so.6(cfree+0x22) [0x7fb1d62ab292]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(+0x76851c) [0x7fb1cba1451c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(std::_Sp_counted_ptr_inplace<lsst::afw::detection::PeakRecord, std::allocatorlsst::afw::detection::PeakRecord, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x56) [0x7fb1cb5696f6]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/7.0-hsc+1/lib/libmeas_algorithms.so(std::_Sp_counted_ptr_inplace<lsst::afw::detection::Footprint, std::allocatorlsst::afw::detection::Footprint, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x73) [0x7fb1cc219473]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/detection/footprint.so(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x46) [0x7fb1835aa4d6]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/detection/footprint.so(+0x1f1640) [0x7fb1835aa640]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc+1/python/lsst/ip/isr/applyLookupTable.so(+0x1f5208) [0x7fb1cc8d7208]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc+1/python/lsst/ip/isr/applyLookupTable.so(+0x1f5a6e) [0x7fb1cc8d7a6e]
python(+0x10ed42) [0x559c45de6d42]
python((null)+0x48b) [0x559c45dee0ab]
python(_PyEval_EvalCodeWithName+0x48b) [0x559c45dee0ab]
python((null)+0x387) [0x559c45e50337]
python(_PyFunction_FastCallKeywords+0x387) [0x559c45e50337]
python((null)+0x14f5) [0x559c45ea95d5]
python(_PyEval_EvalFrameDefault+0x14f5) [0x559c45ea95d5]
python((null)+0xba9) [0x559c45dee7c9]
python(_PyEval_EvalCodeWithName+0xba9) [0x559c45dee7c9]
python((null)+0x387) [0x559c45e50337]
python(_PyFunction_FastCallKeywords+0x387) [0x559c45e50337]
python((null)+0x535) [0x559c45ea8615]
python(_PyEval_EvalFrameDefault+0x535) [0x559c45ea8615]
python((null)+0x2f9) [0x559c45dedf19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x559c45dedf19]
python((null)+0x3d8) [0x559c45def1e8]
python(_PyFunction_FastCallDict+0x3d8) [0x559c45def1e8]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0x5da) [0x559c45dee1fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x559c45dee1fa]
python((null)+0x3d8) [0x559c45def1e8]
python(_PyFunction_FastCallDict+0x3d8) [0x559c45def1e8]
python((null)+0x63) [0x559c45e05cb3]
python(_PyObject_Call_Prepend+0x63) [0x559c45e05cb3]
python(PyObject_Call+0x6e) [0x559c45dfa7de]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0x10b) [0x559c45deef1b]
python(_PyFunction_FastCallDict+0x10b) [0x559c45deef1b]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0x5da) [0x559c45dee1fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x559c45dee1fa]
python((null)+0x387) [0x559c45e50337]
python(_PyFunction_FastCallKeywords+0x387) [0x559c45e50337]
python((null)+0x535) [0x559c45ea8615]
python(_PyEval_EvalFrameDefault+0x535) [0x559c45ea8615]
python((null)+0x10b) [0x559c45deef1b]
python(_PyFunction_FastCallDict+0x10b) [0x559c45deef1b]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0x5da) [0x559c45dee1fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x559c45dee1fa]
python((null)+0x1d4) [0x559c45deefe4]
python(_PyFunction_FastCallDict+0x1d4) [0x559c45deefe4]
python((null)+0x63) [0x559c45e05cb3]
python(_PyObject_Call_Prepend+0x63) [0x559c45e05cb3]
python(PyObject_Call+0x6e) [0x559c45dfa7de]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0xfb) [0x559c45e500ab]
python(_PyFunction_FastCallKeywords+0xfb) [0x559c45e500ab]
python((null)+0x535) [0x559c45ea8615]
python(_PyEval_EvalFrameDefault+0x535) [0x559c45ea8615]
python((null)+0x10b) [0x559c45deef1b]
python(_PyFunction_FastCallDict+0x10b) [0x559c45deef1b]
python((null)+0x63) [0x559c45e05cb3]
python(_PyObject_Call_Prepend+0x63) [0x559c45e05cb3]
python(+0x170cca) [0x559c45e48cca]
python((null)+0x93) [0x559c45def403]
python(_PyObject_FastCallDict+0x93) [0x559c45def403]
python(+0x14e942) [0x559c45e26942]
python(+0x1bbee4) [0x559c45e93ee4]
python((null)+0x128) [0x559c45e512d8]
python(_PyObject_FastCallKeywords+0x128) [0x559c45e512d8]
python((null)+0x4cb6) [0x559c45eacd96]
python(_PyEval_EvalFrameDefault+0x4cb6) [0x559c45eacd96]
python((null)+0xfb) [0x559c45e500ab]
python(_PyFunction_FastCallKeywords+0xfb) [0x559c45e500ab]
python((null)+0x535) [0x559c45ea8615]
python(_PyEval_EvalFrameDefault+0x535) [0x559c45ea8615]
python((null)+0x2f9) [0x559c45dedf19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x559c45dedf19]
python((null)+0x325) [0x559c45e502d5]
python(_PyFunction_FastCallKeywords+0x325) [0x559c45e502d5]
python((null)+0x4b64) [0x559c45eacc44]
python(_PyEval_EvalFrameDefault+0x4b64) [0x559c45eacc44]
python((null)+0x2f9) [0x559c45dedf19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x559c45dedf19]
python(PyEval_EvalCodeEx+0x44) [0x559c45deedd4]
python(PyEval_EvalCode+0x1c) [0x559c45deedfc]
python(+0x22f9e4) [0x559c45f079e4]
python(PyRun_FileExFlags+0xa1) [0x559c45f11bd1]
python(PyRun_SimpleFileExFlags+0x1c3) [0x559c45f11dc3]
python(+0x23aedb) [0x559c45f12edb]
python((null)+0x3c) [0x559c45f12fbc]
python(_Py_UnixMain+0x3c) [0x559c45f12fbc]
/lib64/libc.so.6((null)+0xf1) [0x7fb1d6245401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7fb1d6245401]
python(+0x1dfed2) [0x559c45eb7ed2]
Segmentation fault (core dumped)

price · June 25, 2019, 1:25am

I’ve verified that HSC processing doesn’t go into applyLookupTable (it uses a different style of linearity correction), and that there are no relevant memory errors identified by valgrind that might cause strange behaviour. I’m therefore at a loss to explain the observed segfault. The only suggestion I can make is that you put a pdb breakpoint in lsst.ip.isr.IsrTask.run and step through to see where the segfault is happening.

Charles · June 25, 2019, 3:48pm

I don’t exactly know how to do that correctly. Where is the file of lsst.ip.isr.IsrTask.run I can edit?

price · June 25, 2019, 4:05pm

After setting up the pipeline, do:

git clone git://github.com/HyperSuprime-Cam/ip_isr.git
cd ip_isr
setup -jr .
scons

Then edit python/lsst/ip/isr/isrTask.py, inserting at the top of the IsrTask's run method:

import pdb; pdb.set_trace()

Then run your command as usual, and you’ll be dumped into the python debugger at the appropriate place. You can step through each instruction with the n or next command or step into an instruction with the s or step command, and you can p or print the values of variables.

Please report the last code line you see at which the segfault occurs, and any other relevant observations.

Charles · June 26, 2019, 4:15am

I am not aware of a obvious trace point in output, maybe you have an idea for that.
Few notes:
(1) That may be not relevant, I have to setup few environment parameters (OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS) to enable the running.
(2) I tested the same command in our different nodes, still got the same message.
(3) I got some warnings when I run scons (see following message)

Global pytest run completed successfully
buildConfig([“doc/doxygen.conf”], [“doc/doxygen.conf.in”])
doxygen /data/home/hscpipe5/ip_isr/doc/doxygen.conf
warning: source examples is not a readable file or directory… skipping.
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/doc/afw.tag:1: warning: Internal inconsistency: scope for class lsst::afw::table::detail::SchemaImpl::MakeItem::apply not found!
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/doc/afw.tag:1: warning: Internal inconsistency: scope for class lsst::afw::table::detail::SchemaMapperImpl::MakeKeyPair::apply not found!
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:335: warning: included file runAssembleTask.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:339: warning: included file exampleUtils.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:346: warning: included file runAssembleTask.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/isrTask.py:853: warning: Invalid list item found
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:335: warning: included file runAssembleTask.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:339: warning: included file exampleUtils.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:346: warning: included file runAssembleTask.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/isrTask.py:853: warning: Invalid list item found
scons: done building targets.

Here is the last few code lines and output message:

–Call–
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/parallel.py(609)parseAndRun()
-> @classmethod
(Pdb) n
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/parallel.py(616)parseAndRun()
-> pool = startPool()
(Pdb) n
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/parallel.py(617)parseAndRun()
-> results = super(BatchParallelTask, cls).parseAndRun(*args, **kwargs)

output:

root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/singleFrameDriver.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/hsc/singleFrameDriver.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp772-core1/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772-core1/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772-core1/CALIB/calibRegistry.sqlite3
root INFO: Running: -c /array/users/ytchen/processed/2019/201904-pi-hp772-core1 --rerun JT-201904-pi-hp772-core1-2 --calib=/array/users/ytchen/processed/2019/201904-pi-hp772-core1/CALIB/ --id visit=174620 --config processCcd.isr.doBias=False processCcd.isr.doDark=False processCcd.isr.doFringe=False processCcd.isr.doAttachTransmissionCurve=False --clobber-config --noExit
root INFO: Processing 112 targets with a pool of 1 processes…
singleFrameDriver INFO: subaru1.asiaa.sinica.edu.tw:21140: Start processing {‘visit’: 174620, ‘field’: ‘JTL504’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
singleFrameDriver.processCcd INFO: Processing {‘visit’: 174620, ‘field’: ‘JTL504’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
singleFrameDriver.processCcd.isr INFO: Performing ISR on sensor {‘visit’: 174620, ‘field’: ‘JTL504’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
singleFrameDriver.processCcd.isr INFO: Converting exposure to floating point values
singleFrameDriver.processCcd.isr WARN: Amplifier 1 is bad.
singleFrameDriver.processCcd.isr INFO: Assembling CCD from amplifiers
singleFrameDriver.processCcd.isr INFO: Applying linearizer.
singleFrameDriver.processCcd.isr INFO: Applying crosstalk correction.
singleFrameDriver.processCcd.isr.crosstalk INFO: Applying crosstalk correction/Subaru
singleFrameDriver.processCcd.isr INFO: Widening saturation trails.
singleFrameDriver.processCcd.isr INFO: Applying brighter fatter correction.
singleFrameDriver.processCcd.isr INFO: Applying flat correction.
singleFrameDriver.processCcd.isr INFO: Setting rough magnitude zero point: 32.692803
singleFrameDriver.processCcd.isr INFO: Constructing Vignette polygon.
singleFrameDriver.processCcd.isr INFO: Adding a distortion model to the WCS.
singleFrameDriver.processCcd.isr INFO: Measuring background level:
singleFrameDriver.processCcd.isr INFO: Flattened sky level: 312.062653 +/- 10.555221
singleFrameDriver.processCcd.isr INFO: Measuring sky levels in 8x16 grids: 312.384396
singleFrameDriver.processCcd.isr INFO: Sky flatness in 8x16 grids - pp: 0.015111 rms: 0.003671
singleFrameDriver.processCcd.charImage INFO: Processing {‘visit’: 174620, ‘field’: ‘JTL504’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
singleFrameDriver.processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
singleFrameDriver.processCcd.charImage.repair INFO: Identified 32 cosmic rays.
singleFrameDriver.processCcd.charImage.detection INFO: Detected 371 positive peaks in 35 footprints and 0 negative peaks in 0 footprints to 50 sigma
singleFrameDriver.processCcd.charImage.detection INFO: Resubtracting the background after object detection
singleFrameDriver.processCcd.charImage.measurement INFO: Measuring 35 sources (35 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7f93735dfbd4]
/lib64/libc.so.6(+0x35950) [0x7f93811f3950]
/lib64/libc.so.6(+0x16b0e0) [0x7f93813290e0]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(lsst::afw::table::AliasMap::_apply(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) const+0x64) [0x7f93670f6944]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(lsst::afw::table::SchemaItemlsst::afw::table::Flag lsst::afw::table::Schema::findlsst::afw::table::Flag(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const+0x3c) [0x7f936714886c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc+1/lib/libmeas_base.so(lsst::meas::base::SdssCentroidAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0xe3) [0x7f933d4b2803]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc+1/python/lsst/meas/base/sdssCentroid.so(+0x1f9aed) [0x7f931ca6baed]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc+1/python/lsst/meas/base/sdssCentroid.so(+0x1fb897) [0x7f931ca6d897]
python((null)+0x264) [0x55afb23dbe24]
python(_PyMethodDef_RawFastCallKeywords+0x264) [0x55afb23dbe24]
python((null)+0x21) [0x55afb23dbf41]
python(_PyCFunction_FastCallKeywords+0x21) [0x55afb23dbf41]
python((null)+0x5266) [0x55afb2436d26]
python(_PyEval_EvalFrameDefault+0x5266) [0x55afb2436d26]
python((null)+0x10b) [0x55afb23780eb]
python(_PyFunction_FastCallDict+0x10b) [0x55afb23780eb]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x325) [0x55afb23db0f5]
python(_PyFunction_FastCallKeywords+0x325) [0x55afb23db0f5]
python((null)+0x530) [0x55afb2431ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55afb2431ff0]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x3dc) [0x55afb23783bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55afb23783bc]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x3dc) [0x55afb23783bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55afb23783bc]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x3dc) [0x55afb23783bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55afb23783bc]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x3dc) [0x55afb23783bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55afb23783bc]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x10b) [0x55afb23780eb]
python(_PyFunction_FastCallDict+0x10b) [0x55afb23780eb]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x530) [0x55afb2431ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55afb2431ff0]
python((null)+0x10b) [0x55afb23780eb]
python(_PyFunction_FastCallDict+0x10b) [0x55afb23780eb]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0xfb) [0x55afb23daecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55afb23daecb]
python((null)+0x530) [0x55afb2431ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55afb2431ff0]
python((null)+0xfb) [0x55afb23daecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55afb23daecb]
python((null)+0x4aa1) [0x55afb2436561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55afb2436561]
python((null)+0x10b) [0x55afb23780eb]
python(_PyFunction_FastCallDict+0x10b) [0x55afb23780eb]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(+0x17194a) [0x55afb23d394a]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0xab9) [0x55afb2377899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55afb2377899]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x6e6) [0x55afb24321a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55afb24321a6]
python((null)+0xab9) [0x55afb2377899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55afb2377899]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0xab9) [0x55afb2377899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55afb2377899]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python(+0x21aef7) [0x55afb247cef7]
python((null)+0x49b) [0x55afb23dc46b]
python(_PyObject_FastCallKeywords+0x49b) [0x55afb23dc46b]
python((null)+0x5266) [0x55afb2436d26]
python(_PyEval_EvalFrameDefault+0x5266) [0x55afb2436d26]
python((null)+0xfb) [0x55afb23daecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55afb23daecb]
python((null)+0x530) [0x55afb2431ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55afb2431ff0]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x4aa1) [0x55afb2436561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55afb2436561]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python(PyEval_EvalCodeEx+0x44) [0x55afb2377fa4]
python(PyEval_EvalCode+0x1c) [0x55afb2377fcc]
python(+0x22f664) [0x55afb2491664]
python(PyRun_StringFlags+0x7d) [0x55afb249c77d]
python(PyRun_SimpleStringFlags+0x3f) [0x55afb249c7df]
python(+0x23a8dd) [0x55afb249c8dd]
python((null)+0x3c) [0x55afb249cc4c]
python(_Py_UnixMain+0x3c) [0x55afb249cc4c]
/lib64/libc.so.6((null)+0xf1) [0x7f93811de401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7f93811de401]
python(+0x1df982) [0x55afb2441982]
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 21140 RUNNING AT subaru1.asiaa.sinica.edu.tw
= EXIT CODE: 139
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

price · June 26, 2019, 4:34am

Whoa, that’s a different backtrace than the one you posted earlier!

What compiler did you use to build hscPipe?

Charles · June 26, 2019, 9:59am

I ran a different frame/visit with only one core, you can see a similar message in the original post, which used 4 cores in parallel.

I am not sure what is the compiler used to build hscPipe (I had sent an email to the administrator), but ‘scons’ should use gcc 6.3.1 to build the IsrTask.

Part of output:

scons: Reading SConscript files …
EUPS integration: enabled
Checking who built the CC compiler…gcc=6.3.1
CC is gcc version 6.3.1
Checking for C++14 support
Checking whether the C++ compiler works… yes
C++14 supported with ‘-std=c++14’
Setting up environment to build package ‘ip_isr’.
Running pytest with 1 process
pytest: automated test discovery mode enabled.
scons: done reading SConscript files.
scons: Building targets …
…
…
============================= test session starts ==============================
platform linux – Python 3.7.2, pytest-3.6.2, py-1.7.0, pluggy-0.6.0
rootdir: /data/home/hscpipe5/ip_isr, inifile: setup.cfg
plugins: cov-2.5.1, session2file-0.1.9, forked-0.2, xdist-1.20.1, flake8-1.0.4, remotedata-0.3.1, openfiles-0.3.2, doctestplus-0.3.0, arraydiff-0.3

price · June 26, 2019, 2:00pm

Same compiler we’re using at Princeton.

I think the best way to proceed from here would be to get a copy of your data, and see if I can reproduce the problem. I’ll need your calibs and the exposures it’s crashing on.

price · June 26, 2019, 8:16pm

I just ran the data you sent me with processCcd.py and singleFrameDriver.py, and didn’t hit any segfaults. I’ll run a single CCD through with valgrind just to make sure, but I think this must be some sort of problem on your end. I suggest rebuilding the stack from scratch, and trying a different machine.