Bad termination from running singleFrameDriver.py via hscPipe v7.5


(Ying-Tung Chen) #1

Hi,

I ran the singleFrameDriver.py after I use ingestCalibs.py to create calibs as Paul’s suggestion. The process looks OK in the begining, but it was terminated by EXIT CODE: 139. It doesn’t look like a problem caused by calibration setting. Here is the full command I executed:
singleFrameDriver.py $WORKDIR --rerun $RERUNFRAME --calib=$WORKDIR/CALIB/ --id visit=174618 --config processCcd.isr.doBias=False processCcd.isr.doDark=False processCcd.isr.doFringe=False processCcd.isr.doAttachTransmissionCurve=False processCcd.isr.doBrighterFatter=False --job reduceFrames.g.JTL503.hp5 --cores $RCHP5 --clobber-config

This is full log that display on the screen:

root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.2-hsc/config/singleFrameDriver.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.2-hsc/config/hsc/singleFrameDriver.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.2-hsc/config/singleFrameDriver.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.2-hsc/config/hsc/singleFrameDriver.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
root INFO: Running: -c /array/users/ytchen/processed/2019/201904-pi-hp75 --rerun JT-201904-pi-hp75 --calib=/array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/ --id visit=174618 --config processCcd.isr.doBias=False processCcd.isr.doDark=False processCcd.isr.doFringe=False processCcd.isr.doAttachTransmissionCurve=False processCcd.isr.doBrighterFatter=False --clobber-config --noExit
root INFO: Processing 112 targets with a pool of 4 processes…
139623 INFO 2019-06-11T13:38:51.188+0800 CameraMapper: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
139623 INFO 2019-06-11T13:38:51.190+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139622 INFO 2019-06-11T13:38:51.197+0800 CameraMapper: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
139622 INFO 2019-06-11T13:38:51.199+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139621 INFO 2019-06-11T13:38:51.205+0800 CameraMapper: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp75/registry.sqlite3
139621 INFO 2019-06-11T13:38:51.206+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139623 INFO 2019-06-11T13:38:52.268+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139622 INFO 2019-06-11T13:38:52.299+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139621 INFO 2019-06-11T13:38:52.301+0800 CameraMapper: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp75/CALIB/calibRegistry.sqlite3
139622 INFO 2019-06-11T13:38:53.166+0800 singleFrameDriver: ichiro.asiaa.sinica.edu.tw:139622: Start processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 1, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139622 INFO 2019-06-11T13:38:53.166+0800 singleFrameDriver.processCcd: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 1, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139622 INFO 2019-06-11T13:38:53.167+0800 singleFrameDriver.processCcd.isr: Performing ISR on sensor {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 1, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 INFO 2019-06-11T13:38:53.174+0800 singleFrameDriver: ichiro.asiaa.sinica.edu.tw:139621: Start processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 INFO 2019-06-11T13:38:53.175+0800 singleFrameDriver.processCcd: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 INFO 2019-06-11T13:38:53.175+0800 singleFrameDriver.processCcd.isr: Performing ISR on sensor {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139623 INFO 2019-06-11T13:38:53.206+0800 singleFrameDriver: ichiro.asiaa.sinica.edu.tw:139623: Start processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 2, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139623 INFO 2019-06-11T13:38:53.206+0800 singleFrameDriver.processCcd: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 2, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139623 INFO 2019-06-11T13:38:53.206+0800 singleFrameDriver.processCcd.isr: Performing ISR on sensor {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 2, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 INFO 2019-06-11T13:38:55.278+0800 singleFrameDriver.processCcd.isr: Converting exposure to floating point values
139621 WARN 2019-06-11T13:38:55.450+0800 singleFrameDriver.processCcd.isr: Amplifier 1 is bad.
139622 INFO 2019-06-11T13:38:55.461+0800 singleFrameDriver.processCcd.isr: Converting exposure to floating point values
139623 INFO 2019-06-11T13:38:55.495+0800 singleFrameDriver.processCcd.isr: Converting exposure to floating point values
139621 INFO 2019-06-11T13:38:55.552+0800 singleFrameDriver.processCcd.isr: Assembling CCD from amplifiers
139622 INFO 2019-06-11T13:38:55.733+0800 singleFrameDriver.processCcd.isr: Assembling CCD from amplifiers
139623 INFO 2019-06-11T13:38:55.783+0800 singleFrameDriver.processCcd.isr: Assembling CCD from amplifiers
139621 INFO 2019-06-11T13:38:56.102+0800 singleFrameDriver.processCcd.isr: Applying linearizer.
139621 INFO 2019-06-11T13:38:56.138+0800 singleFrameDriver.processCcd.isr: Applying crosstalk correction.
139621 INFO 2019-06-11T13:38:56.138+0800 singleFrameDriver.processCcd.isr.crosstalk: Applying crosstalk correction/Subaru
139622 INFO 2019-06-11T13:38:56.228+0800 singleFrameDriver.processCcd.isr: Applying linearizer.
139622 INFO 2019-06-11T13:38:56.255+0800 singleFrameDriver.processCcd.isr: Applying crosstalk correction.
139622 INFO 2019-06-11T13:38:56.255+0800 singleFrameDriver.processCcd.isr.crosstalk: Applying crosstalk correction/Subaru
139623 INFO 2019-06-11T13:38:56.307+0800 singleFrameDriver.processCcd.isr: Applying linearizer.
139623 INFO 2019-06-11T13:38:56.336+0800 singleFrameDriver.processCcd.isr: Applying crosstalk correction.
139623 INFO 2019-06-11T13:38:56.336+0800 singleFrameDriver.processCcd.isr.crosstalk: Applying crosstalk correction/Subaru
139621 INFO 2019-06-11T13:38:57.049+0800 singleFrameDriver.processCcd.isr: Widening saturation trails.
139621 INFO 2019-06-11T13:38:57.065+0800 singleFrameDriver.processCcd.isr: Applying flat correction.
139621 INFO 2019-06-11T13:38:57.121+0800 singleFrameDriver.processCcd.isr: Masking and interpolating defects.
139622 INFO 2019-06-11T13:38:57.133+0800 singleFrameDriver.processCcd.isr: Widening saturation trails.
139622 INFO 2019-06-11T13:38:57.159+0800 singleFrameDriver.processCcd.isr: Applying flat correction.
139623 INFO 2019-06-11T13:38:57.201+0800 singleFrameDriver.processCcd.isr: Widening saturation trails.
139622 INFO 2019-06-11T13:38:57.214+0800 singleFrameDriver.processCcd.isr: Masking and interpolating defects.
139623 INFO 2019-06-11T13:38:57.215+0800 singleFrameDriver.processCcd.isr: Applying flat correction.
139623 INFO 2019-06-11T13:38:57.264+0800 singleFrameDriver.processCcd.isr: Masking and interpolating defects.
139621 INFO 2019-06-11T13:38:57.516+0800 singleFrameDriver.processCcd.isr: Interpolating saturated pixels.
139622 INFO 2019-06-11T13:38:57.676+0800 singleFrameDriver.processCcd.isr: Interpolating saturated pixels.
139623 INFO 2019-06-11T13:38:57.687+0800 singleFrameDriver.processCcd.isr: Interpolating saturated pixels.
139621 INFO 2019-06-11T13:38:57.968+0800 singleFrameDriver.processCcd.isr: Masking and interpolating NAN value pixels.
139623 INFO 2019-06-11T13:38:58.136+0800 singleFrameDriver.processCcd.isr: Masking and interpolating NAN value pixels.
139622 INFO 2019-06-11T13:38:58.144+0800 singleFrameDriver.processCcd.isr: Masking and interpolating NAN value pixels.
139621 INFO 2019-06-11T13:38:58.256+0800 singleFrameDriver.processCcd.isr: Setting rough magnitude zero point: 32.692803
139621 INFO 2019-06-11T13:38:58.256+0800 singleFrameDriver.processCcd.isr: Constructing Vignette polygon.
139621 INFO 2019-06-11T13:38:58.279+0800 singleFrameDriver.processCcd.isr: Adding a distortion model to the WCS.
139621 INFO 2019-06-11T13:38:58.281+0800 singleFrameDriver.processCcd.isr: Measuring background level:
139623 INFO 2019-06-11T13:38:58.460+0800 singleFrameDriver.processCcd.isr: Setting rough magnitude zero point: 32.692803
139623 INFO 2019-06-11T13:38:58.460+0800 singleFrameDriver.processCcd.isr: Constructing Vignette polygon.
139623 INFO 2019-06-11T13:38:58.461+0800 singleFrameDriver.processCcd.isr: Adding a distortion model to the WCS.
139623 INFO 2019-06-11T13:38:58.464+0800 singleFrameDriver.processCcd.isr: Measuring background level:
139622 INFO 2019-06-11T13:38:58.472+0800 singleFrameDriver.processCcd.isr: Setting rough magnitude zero point: 32.692803
139622 INFO 2019-06-11T13:38:58.472+0800 singleFrameDriver.processCcd.isr: Constructing Vignette polygon.
139622 INFO 2019-06-11T13:38:58.473+0800 singleFrameDriver.processCcd.isr: Adding a distortion model to the WCS.
139622 INFO 2019-06-11T13:38:58.476+0800 singleFrameDriver.processCcd.isr: Measuring background level:
139621 INFO 2019-06-11T13:38:58.585+0800 singleFrameDriver.processCcd.isr: Flattened sky level: 305.868256 +/- 10.445378
139623 INFO 2019-06-11T13:38:58.808+0800 singleFrameDriver.processCcd.isr: Flattened sky level: 304.669098 +/- 10.142621
139622 INFO 2019-06-11T13:38:58.831+0800 singleFrameDriver.processCcd.isr: Flattened sky level: 306.349213 +/- 9.998964
139621 INFO 2019-06-11T13:38:58.852+0800 singleFrameDriver.processCcd.isr: Measuring sky levels in 8x16 grids: 306.108966
139621 INFO 2019-06-11T13:38:58.852+0800 singleFrameDriver.processCcd.isr: Sky flatness in 8x16 grids - pp: 0.017215 rms: 0.003908
139623 INFO 2019-06-11T13:38:59.146+0800 singleFrameDriver.processCcd.isr: Measuring sky levels in 8x16 grids: 304.970223
139623 INFO 2019-06-11T13:38:59.146+0800 singleFrameDriver.processCcd.isr: Sky flatness in 8x16 grids - pp: 0.034628 rms: 0.004833
139622 INFO 2019-06-11T13:38:59.172+0800 singleFrameDriver.processCcd.isr: Measuring sky levels in 8x16 grids: 306.399288
139622 INFO 2019-06-11T13:38:59.172+0800 singleFrameDriver.processCcd.isr: Sky flatness in 8x16 grids - pp: 0.023681 rms: 0.004674
139621 INFO 2019-06-11T13:38:59.186+0800 singleFrameDriver.processCcd.charImage: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139623 INFO 2019-06-11T13:38:59.498+0800 singleFrameDriver.processCcd.charImage: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 2, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139622 INFO 2019-06-11T13:38:59.525+0800 singleFrameDriver.processCcd.charImage: Processing {‘visit’: 174618, ‘field’: ‘JTL503’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 1, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
139621 WARN 2019-06-11T13:38:59.694+0800 singleFrameDriver.processCcd.charImage: Source catalog detected and measured with placeholder or default PSF
139621 INFO 2019-06-11T13:39:00.075+0800 singleFrameDriver.processCcd.charImage.repair: Identified 42 cosmic rays.
139623 WARN 2019-06-11T13:39:00.090+0800 singleFrameDriver.processCcd.charImage: Source catalog detected and measured with placeholder or default PSF
139622 WARN 2019-06-11T13:39:00.135+0800 singleFrameDriver.processCcd.charImage: Source catalog detected and measured with placeholder or default PSF
139623 INFO 2019-06-11T13:39:00.478+0800 singleFrameDriver.processCcd.charImage.repair: Identified 69 cosmic rays.
139622 INFO 2019-06-11T13:39:00.496+0800 singleFrameDriver.processCcd.charImage.repair: Identified 78 cosmic rays.
139621 INFO 2019-06-11T13:39:01.319+0800 singleFrameDriver.processCcd.charImage.detection: Detected 129 positive peaks in 35 footprints and 0 negative peaks in 0 footprints to 50 sigma
139621 INFO 2019-06-11T13:39:01.565+0800 singleFrameDriver.processCcd.charImage.detection: Resubtracting the background after object detection
139622 INFO 2019-06-11T13:39:01.799+0800 singleFrameDriver.processCcd.charImage.detection: Detected 341 positive peaks in 54 footprints and 0 negative peaks in 0 footprints to 50 sigma
139623 INFO 2019-06-11T13:39:01.855+0800 singleFrameDriver.processCcd.charImage.detection: Detected 256 positive peaks in 58 footprints and 0 negative peaks in 0 footprints to 50 sigma
139622 INFO 2019-06-11T13:39:02.133+0800 singleFrameDriver.processCcd.charImage.detection: Resubtracting the background after object detection
139623 INFO 2019-06-11T13:39:02.190+0800 singleFrameDriver.processCcd.charImage.detection: Resubtracting the background after object detection
139621 INFO 2019-06-11T13:39:02.348+0800 singleFrameDriver.processCcd.charImage.measurement: Measuring 35 sources (35 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7f843fbaabd4]
/lib64/libc.so.6(+0x35990) [0x7f845611d990]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.2-hsc/lib/libafw.so(lsst::afw::table::AliasMap::_apply(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) const+0x78) [0x7f84336a7958]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.2-hsc/lib/libafw.so(lsst::afw::table::SchemaItemlsst::afw::table::Flag lsst::afw::table::Schema::findlsst::afw::table::Flag(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const+0x3c) [0x7f84336f986c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::meas::base::SdssCentroidAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0xe3) [0x7f8409a5e803]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1f9aed) [0x7f83e9017aed]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1fb897) [0x7f83e9019897]
python((null)+0x264) [0x557bb7abde24]
python(_PyMethodDef_RawFastCallKeywords+0x264) [0x557bb7abde24]
python((null)+0x21) [0x557bb7abdf41]
python(_PyCFunction_FastCallKeywords+0x21) [0x557bb7abdf41]
python((null)+0x5266) [0x557bb7b18d26]
python(_PyEval_EvalFrameDefault+0x5266) [0x557bb7b18d26]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x325) [0x557bb7abd0f5]
python(_PyFunction_FastCallKeywords+0x325) [0x557bb7abd0f5]
python((null)+0x530) [0x557bb7b13ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x557bb7b13ff0]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x3dc) [0x557bb7a5a3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x557bb7a5a3bc]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x3dc) [0x557bb7a5a3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x557bb7a5a3bc]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x3dc) [0x557bb7a5a3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x557bb7a5a3bc]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x3dc) [0x557bb7a5a3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x557bb7a5a3bc]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x14e1) [0x557bb7b14fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x557bb7b14fa1]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x530) [0x557bb7b13ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x557bb7b13ff0]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0xfb) [0x557bb7abcecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x557bb7abcecb]
python((null)+0x530) [0x557bb7b13ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x557bb7b13ff0]
python((null)+0xfb) [0x557bb7abcecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x557bb7abcecb]
python((null)+0x4aa1) [0x557bb7b18561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x557bb7b18561]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(+0x17194a) [0x557bb7ab594a]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0xab9) [0x557bb7a59899]
python(_PyEval_EvalCodeWithName+0xab9) [0x557bb7a59899]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x6e6) [0x557bb7b141a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x557bb7b141a6]
python((null)+0xab9) [0x557bb7a59899]
python(_PyEval_EvalCodeWithName+0xab9) [0x557bb7a59899]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x63) [0x557bb7a71473]
python(_PyObject_Call_Prepend+0x63) [0x557bb7a71473]
python(PyObject_Call+0x6e) [0x557bb7a6619e]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x10b) [0x557bb7a5a0eb]
python(_PyFunction_FastCallDict+0x10b) [0x557bb7a5a0eb]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x6e6) [0x557bb7b141a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x557bb7b141a6]
python((null)+0xab9) [0x557bb7a59899]
python(_PyEval_EvalCodeWithName+0xab9) [0x557bb7a59899]
python((null)+0x1d4) [0x557bb7a5a1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x557bb7a5a1b4]
python((null)+0x1f3f) [0x557bb7b159ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x557bb7b159ff]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x530) [0x557bb7b13ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x557bb7b13ff0]
python(+0x178d3b) [0x557bb7abcd3b]
python((null)+0x6e6) [0x557bb7b141a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x557bb7b141a6]
python((null)+0x5da) [0x557bb7a593ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x557bb7a593ba]
python((null)+0x387) [0x557bb7abd157]
python(_PyFunction_FastCallKeywords+0x387) [0x557bb7abd157]
python((null)+0x4aa1) [0x557bb7b18561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x557bb7b18561]
python((null)+0x2f9) [0x557bb7a590d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x557bb7a590d9]
python(PyEval_EvalCodeEx+0x44) [0x557bb7a59fa4]
python(PyEval_EvalCode+0x1c) [0x557bb7a59fcc]
python(+0x22f664) [0x557bb7b73664]
python(PyRun_StringFlags+0x7d) [0x557bb7b7e77d]
python(PyRun_SimpleStringFlags+0x3f) [0x557bb7b7e7df]
python(+0x23a8dd) [0x557bb7b7e8dd]
python((null)+0x3c) [0x557bb7b7ec4c]
python(_Py_UnixMain+0x3c) [0x557bb7b7ec4c]
/lib64/libc.so.6((null)+0xf1) [0x7f8456108401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7f8456108401]
python(+0x1df982) [0x557bb7b23982]
139622 INFO 2019-06-11T13:39:02.664+0800 singleFrameDriver.processCcd.charImage.measurement: Measuring 54 sources (54 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7f296f986bd4]
/lib64/libc.so.6(+0x35990) [0x7f2985ef9990]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::afw::table::SourceRecord::getCentroidFlag() const+0x5d) [0x7f293982078d]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::meas::base::SafeCentroidExtractor::operator()(lsst::afw::table::SourceRecord&, lsst::meas::base::FlagHandler const&) const+0x2a3) [0x7f293981ff53]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::meas::base::SdssCentroidAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0x48) [0x7f293983a768]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1f9aed) [0x7f2918df3aed]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1fb897) [0x7f2918df5897]
python((null)+0x264) [0x55b73a541e24]
python(_PyMethodDef_RawFastCallKeywords+0x264) [0x55b73a541e24]
python((null)+0x21) [0x55b73a541f41]
python(_PyCFunction_FastCallKeywords+0x21) [0x55b73a541f41]
python((null)+0x5266) [0x55b73a59cd26]
python(_PyEval_EvalFrameDefault+0x5266) [0x55b73a59cd26]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x325) [0x55b73a5410f5]
python(_PyFunction_FastCallKeywords+0x325) [0x55b73a5410f5]
python((null)+0x530) [0x55b73a597ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b73a597ff0]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x3dc) [0x55b73a4de3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b73a4de3bc]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x3dc) [0x55b73a4de3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b73a4de3bc]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x3dc) [0x55b73a4de3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b73a4de3bc]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x3dc) [0x55b73a4de3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b73a4de3bc]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x14e1) [0x55b73a598fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b73a598fa1]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x530) [0x55b73a597ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b73a597ff0]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0xfb) [0x55b73a540ecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55b73a540ecb]
python((null)+0x530) [0x55b73a597ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b73a597ff0]
python((null)+0xfb) [0x55b73a540ecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55b73a540ecb]
python((null)+0x4aa1) [0x55b73a59c561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55b73a59c561]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(+0x17194a) [0x55b73a53994a]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0xab9) [0x55b73a4dd899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b73a4dd899]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x6e6) [0x55b73a5981a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b73a5981a6]
python((null)+0xab9) [0x55b73a4dd899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b73a4dd899]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x63) [0x55b73a4f5473]
python(_PyObject_Call_Prepend+0x63) [0x55b73a4f5473]
python(PyObject_Call+0x6e) [0x55b73a4ea19e]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x10b) [0x55b73a4de0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b73a4de0eb]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x6e6) [0x55b73a5981a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b73a5981a6]
python((null)+0xab9) [0x55b73a4dd899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b73a4dd899]
python((null)+0x1d4) [0x55b73a4de1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b73a4de1b4]
python((null)+0x1f3f) [0x55b73a5999ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b73a5999ff]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x530) [0x55b73a597ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b73a597ff0]
python(+0x178d3b) [0x55b73a540d3b]
python((null)+0x6e6) [0x55b73a5981a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b73a5981a6]
python((null)+0x5da) [0x55b73a4dd3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b73a4dd3ba]
python((null)+0x387) [0x55b73a541157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b73a541157]
python((null)+0x4aa1) [0x55b73a59c561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55b73a59c561]
python((null)+0x2f9) [0x55b73a4dd0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b73a4dd0d9]
python(PyEval_EvalCodeEx+0x44) [0x55b73a4ddfa4]
python(PyEval_EvalCode+0x1c) [0x55b73a4ddfcc]
python(+0x22f664) [0x55b73a5f7664]
python(PyRun_StringFlags+0x7d) [0x55b73a60277d]
python(PyRun_SimpleStringFlags+0x3f) [0x55b73a6027df]
python(+0x23a8dd) [0x55b73a6028dd]
python((null)+0x3c) [0x55b73a602c4c]
python(_Py_UnixMain+0x3c) [0x55b73a602c4c]
/lib64/libc.so.6((null)+0xf1) [0x7f2985ee4401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7f2985ee4401]
python(+0x1df982) [0x55b73a5a7982]
139623 INFO 2019-06-11T13:39:02.697+0800 singleFrameDriver.processCcd.charImage.measurement: Measuring 58 sources (58 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7f17e8ccbbd4]
/lib64/libc.so.6(+0x35990) [0x7f17ff23e990]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.2-hsc/lib/libafw.so(lsst::afw::table::AliasMap::_apply(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) const+0x78) [0x7f17dc7c8958]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.2-hsc/lib/libafw.so(lsst::afw::table::SchemaItemlsst::afw::table::Flag lsst::afw::table::Schema::findlsst::afw::table::Flag(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const+0x3c) [0x7f17dc81a86c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/lib/libmeas_base.so(lsst::meas::base::SdssCentroidAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0xe3) [0x7f17b2b7f803]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1f9aed) [0x7f1792138aed]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc/python/lsst/meas/base/sdssCentroid.so(+0x1fb897) [0x7f179213a897]
python((null)+0x264) [0x55b06c52ee24]
python(_PyMethodDef_RawFastCallKeywords+0x264) [0x55b06c52ee24]
python((null)+0x21) [0x55b06c52ef41]
python(_PyCFunction_FastCallKeywords+0x21) [0x55b06c52ef41]
python((null)+0x5266) [0x55b06c589d26]
python(_PyEval_EvalFrameDefault+0x5266) [0x55b06c589d26]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x325) [0x55b06c52e0f5]
python(_PyFunction_FastCallKeywords+0x325) [0x55b06c52e0f5]
python((null)+0x530) [0x55b06c584ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b06c584ff0]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x3dc) [0x55b06c4cb3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b06c4cb3bc]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x3dc) [0x55b06c4cb3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b06c4cb3bc]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x3dc) [0x55b06c4cb3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b06c4cb3bc]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x3dc) [0x55b06c4cb3bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55b06c4cb3bc]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x14e1) [0x55b06c585fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55b06c585fa1]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x530) [0x55b06c584ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b06c584ff0]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0xfb) [0x55b06c52decb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55b06c52decb]
python((null)+0x530) [0x55b06c584ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b06c584ff0]
python((null)+0xfb) [0x55b06c52decb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55b06c52decb]
python((null)+0x4aa1) [0x55b06c589561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55b06c589561]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(+0x17194a) [0x55b06c52694a]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0xab9) [0x55b06c4ca899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b06c4ca899]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x6e6) [0x55b06c5851a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b06c5851a6]
python((null)+0xab9) [0x55b06c4ca899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b06c4ca899]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x63) [0x55b06c4e2473]
python(_PyObject_Call_Prepend+0x63) [0x55b06c4e2473]
python(PyObject_Call+0x6e) [0x55b06c4d719e]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x10b) [0x55b06c4cb0eb]
python(_PyFunction_FastCallDict+0x10b) [0x55b06c4cb0eb]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x6e6) [0x55b06c5851a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b06c5851a6]
python((null)+0xab9) [0x55b06c4ca899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55b06c4ca899]
python((null)+0x1d4) [0x55b06c4cb1b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55b06c4cb1b4]
python((null)+0x1f3f) [0x55b06c5869ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55b06c5869ff]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x530) [0x55b06c584ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55b06c584ff0]
python(+0x178d3b) [0x55b06c52dd3b]
python((null)+0x6e6) [0x55b06c5851a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55b06c5851a6]
python((null)+0x5da) [0x55b06c4ca3ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55b06c4ca3ba]
python((null)+0x387) [0x55b06c52e157]
python(_PyFunction_FastCallKeywords+0x387) [0x55b06c52e157]
python((null)+0x4aa1) [0x55b06c589561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55b06c589561]
python((null)+0x2f9) [0x55b06c4ca0d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55b06c4ca0d9]
python(PyEval_EvalCodeEx+0x44) [0x55b06c4cafa4]
python(PyEval_EvalCode+0x1c) [0x55b06c4cafcc]
python(+0x22f664) [0x55b06c5e4664]
python(PyRun_StringFlags+0x7d) [0x55b06c5ef77d]
python(PyRun_SimpleStringFlags+0x3f) [0x55b06c5ef7df]
python(+0x23a8dd) [0x55b06c5ef8dd]
python((null)+0x3c) [0x55b06c5efc4c]
python(_Py_UnixMain+0x3c) [0x55b06c5efc4c]
/lib64/libc.so.6((null)+0xf1) [0x7f17ff229401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7f17ff229401]
python(+0x1df982) [0x55b06c594982]
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 139622 RUNNING AT ichiro.asiaa.sinica.edu.tw
= EXIT CODE: 139
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions


(K-T Lim) #2

That’s a very odd place to get a segmentation fault, as it’s doing standard C++ library manipulation of standard C++ strings (using a standard C++ map). Segfaults often occur because of a mismatch in versions between pieces of code. Are you sure that the libc you are using is the one the stack was built with? Are there other potential incompatibilities?


(Ying-Tung Chen) #3

Thanks, that’s is a good point to shooting this problem. I do remember that our administrator encountered some incompatible issues during installation. I will check the library with the administrator first.


(Ying-Tung Chen) #4

Our administrator reply that he followed all the instruction of the installation and have no compatibility error/issue during the installation process. I had tried v7.4, but got the same termination. I only see one version libc in the same directory (/lib64/), which was linked to libc-2.24.so. I am really confused by this issue…


(Ying-Tung Chen) #5

Is there a possible solution for this issue? Or is there anyone installed >v7.5, then successfully processed a frame with singleFrameDriver.py?


(Paul Price) #6

Sorry for the late response; I didn’t see this thread.

We have processed a lot of data with hscPipe 7 for the HSC SSP, so I’m inclined to agree with @ktl that the installation and setup is the first place to look. Perhaps you could post exactly how the pipeline was installed? Most importantly, are you using the new binary install?

Do you get the same problem when you use processCcd.py instead of singleFrameDriver.py?


(Ying-Tung Chen) #7

I had wrote an email to ask administrator about the exactly steps of installation.

In the meantime, I tested processCcd.py with following parameters, then got the same problem:
processCcd.py $WORKDIR --rerun $RERUNFRAME --calib=$WORKDIR/CALIB/ --id visit=175292 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False --clobber-config

Output message:

root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/processCcd.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/hsc/processCcd.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp772/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/calibRegistry.sqlite3
root INFO: Running: /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.0-hsc+1/bin/processCcd.py /array/users/ytchen/processed/2019/201904-pi-hp772 --rerun JT-201904-pi-hp772 --calib=/array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/ --id visit=175292 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False --clobber-config
processCcd INFO: Processing {‘visit’: 175292, ‘field’: ‘JTL505’, ‘dateObs’: ‘2019-04-11’, ‘pointing’: 2657, ‘filter’: ‘HSC-I2’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-11’, ‘expTime’: 24.22}
processCcd.isr INFO: Performing ISR on sensor {‘visit’: 175292, ‘field’: ‘JTL505’, ‘dateObs’: ‘2019-04-11’, ‘pointing’: 2657, ‘filter’: ‘HSC-I2’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-11’, ‘expTime’: 24.22}
processCcd.isr INFO: Converting exposure to floating point values
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7fdb32d34bd4]
/lib64/libc.so.6(+0x35990) [0x7fdb4093a990]
/lib64/libc.so.6(cfree+0x22) [0x7fdb4098b292]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(+0x76851c) [0x7fdb360f451c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(std::_Sp_counted_ptr_inplace<lsst::afw::detection::PeakRecord, std::allocatorlsst::afw::detection::PeakRecord, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x56) [0x7fdb35c496f6]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/7.0-hsc+1/lib/libmeas_algorithms.so(std::_Sp_counted_ptr_inplace<lsst::afw::detection::Footprint, std::allocatorlsst::afw::detection::Footprint, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x73) [0x7fdb368f9473]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/detection/footprint.so(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x46) [0x7fdaedc8a4d6]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/detection/footprint.so(+0x1f1640) [0x7fdaedc8a640]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc+1/python/lsst/ip/isr/applyLookupTable.so(+0x1f5208) [0x7fdb36fb7208]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc+1/python/lsst/ip/isr/applyLookupTable.so(+0x1f5a6e) [0x7fdb36fb7a6e]
python(+0x10ed42) [0x5640d5ea9d42]
python((null)+0x48b) [0x5640d5eb10ab]
python(_PyEval_EvalCodeWithName+0x48b) [0x5640d5eb10ab]
python((null)+0x387) [0x5640d5f13337]
python(_PyFunction_FastCallKeywords+0x387) [0x5640d5f13337]
python((null)+0x14f5) [0x5640d5f6c5d5]
python(_PyEval_EvalFrameDefault+0x14f5) [0x5640d5f6c5d5]
python((null)+0xba9) [0x5640d5eb17c9]
python(_PyEval_EvalCodeWithName+0xba9) [0x5640d5eb17c9]
python((null)+0x387) [0x5640d5f13337]
python(_PyFunction_FastCallKeywords+0x387) [0x5640d5f13337]
python((null)+0x535) [0x5640d5f6b615]
python(_PyEval_EvalFrameDefault+0x535) [0x5640d5f6b615]
python((null)+0x2f9) [0x5640d5eb0f19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x5640d5eb0f19]
python((null)+0x3d8) [0x5640d5eb21e8]
python(_PyFunction_FastCallDict+0x3d8) [0x5640d5eb21e8]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0x5da) [0x5640d5eb11fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x5640d5eb11fa]
python((null)+0x3d8) [0x5640d5eb21e8]
python(_PyFunction_FastCallDict+0x3d8) [0x5640d5eb21e8]
python((null)+0x63) [0x5640d5ec8cb3]
python(_PyObject_Call_Prepend+0x63) [0x5640d5ec8cb3]
python(PyObject_Call+0x6e) [0x5640d5ebd7de]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0x10b) [0x5640d5eb1f1b]
python(_PyFunction_FastCallDict+0x10b) [0x5640d5eb1f1b]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0x5da) [0x5640d5eb11fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x5640d5eb11fa]
python((null)+0x387) [0x5640d5f13337]
python(_PyFunction_FastCallKeywords+0x387) [0x5640d5f13337]
python((null)+0x535) [0x5640d5f6b615]
python(_PyEval_EvalFrameDefault+0x535) [0x5640d5f6b615]
python((null)+0x10b) [0x5640d5eb1f1b]
python(_PyFunction_FastCallDict+0x10b) [0x5640d5eb1f1b]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0x5da) [0x5640d5eb11fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x5640d5eb11fa]
python((null)+0x1d4) [0x5640d5eb1fe4]
python(_PyFunction_FastCallDict+0x1d4) [0x5640d5eb1fe4]
python((null)+0x63) [0x5640d5ec8cb3]
python(_PyObject_Call_Prepend+0x63) [0x5640d5ec8cb3]
python(PyObject_Call+0x6e) [0x5640d5ebd7de]
python((null)+0x1e3e) [0x5640d5f6cf1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x5640d5f6cf1e]
python((null)+0xfb) [0x5640d5f130ab]
python(_PyFunction_FastCallKeywords+0xfb) [0x5640d5f130ab]
python((null)+0x535) [0x5640d5f6b615]
python(_PyEval_EvalFrameDefault+0x535) [0x5640d5f6b615]
python((null)+0x10b) [0x5640d5eb1f1b]
python(_PyFunction_FastCallDict+0x10b) [0x5640d5eb1f1b]
python((null)+0x63) [0x5640d5ec8cb3]
python(_PyObject_Call_Prepend+0x63) [0x5640d5ec8cb3]
python(+0x170cca) [0x5640d5f0bcca]
python((null)+0x93) [0x5640d5eb2403]
python(_PyObject_FastCallDict+0x93) [0x5640d5eb2403]
python(+0x14e942) [0x5640d5ee9942]
python(+0x1bbee4) [0x5640d5f56ee4]
python((null)+0x128) [0x5640d5f142d8]
python(_PyObject_FastCallKeywords+0x128) [0x5640d5f142d8]
python((null)+0x4cb6) [0x5640d5f6fd96]
python(_PyEval_EvalFrameDefault+0x4cb6) [0x5640d5f6fd96]
python((null)+0xfb) [0x5640d5f130ab]
python(_PyFunction_FastCallKeywords+0xfb) [0x5640d5f130ab]
python((null)+0x535) [0x5640d5f6b615]
python(_PyEval_EvalFrameDefault+0x535) [0x5640d5f6b615]
python((null)+0x2f9) [0x5640d5eb0f19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x5640d5eb0f19]
python((null)+0x325) [0x5640d5f132d5]
python(_PyFunction_FastCallKeywords+0x325) [0x5640d5f132d5]
python((null)+0x4b64) [0x5640d5f6fc44]
python(_PyEval_EvalFrameDefault+0x4b64) [0x5640d5f6fc44]
python((null)+0x2f9) [0x5640d5eb0f19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x5640d5eb0f19]
python(PyEval_EvalCodeEx+0x44) [0x5640d5eb1dd4]
python(PyEval_EvalCode+0x1c) [0x5640d5eb1dfc]
python(+0x22f9e4) [0x5640d5fca9e4]
python(PyRun_FileExFlags+0xa1) [0x5640d5fd4bd1]
python(PyRun_SimpleFileExFlags+0x1c3) [0x5640d5fd4dc3]
python(+0x23aedb) [0x5640d5fd5edb]
python((null)+0x3c) [0x5640d5fd5fbc]
python(_Py_UnixMain+0x3c) [0x5640d5fd5fbc]
/lib64/libc.so.6((null)+0xf1) [0x7fdb40925401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7fdb40925401]
python(+0x1dfed2) [0x5640d5f7aed2]
Segmentation fault (core dumped)


(K-T Lim) #8

That’s in a different part of the code (and a different version of afw: 7.6 vs. 7.2). Note that the first two lines in the backtrace (libutils.so and libc.so.6(+0x35990)) are the common code that generates the backtrace, so it’s really the third line that is where the problem is occurring. Seems to me something is definitely not right with your installation.


(K-T Lim) #9

And to highlight something @price said, posting the setup command(s) you used or the result of eups list --setup might also help (or at least rule out one source of potential problems).


(Ying-Tung Chen) #10

OK. Our administrator replied that he cannot use the binary install since the OS of the server is Fedora Core. So he installed the hscPipe using eups, instead of binary:

source loadLSST.bash
eups distrib install hscPipe 7.7.2

The following list is the output of eups list --setup:

afw 7.6-hsc setup
apr 1.5.2 current setup
apr_util 1.5.4-1-g78a90a1 current setup
astro_metadata_translator 7.1-hsc setup
astrometry_net 0.67.123ff3e.lsst1+13 current setup
astrometry_net_data 10.0+138 setup
astropy 2.0.1-2-gb0b7760 current setup
astshim 7.3-hsc setup
base 7.0-hsc setup
boost 1.60.lsst1-3-g1c4857aa current setup
cfitsio 3360.lsst5-1-gfc68e3c current setup
coadd_utils 7.0-hsc+1 setup
coord 1.1.lsst+3 current setup
ctrl_pool 7.0-hsc+1 setup
daf_base 7.0-hsc current setup
daf_butler 7.0-hsc+1 setup
daf_persistence 7.0-hsc current setup
dax_ppdb 7.0-hsc+1 setup
display_ds9 7.0-hsc+1 setup
doxygen 1.8.13.lsst1-2-g524ee4e current setup
eigen 7.2-hsc setup
esutil 0.6.2.5.lsst1-1-gdae48e7 current setup
fftw 3.3.4.lsst2 current setup
flake8 3.5.0-3-g1298ec5 current setup
galsim 1.5.1.lsst1-11-gfba72ad current setup
geom 7.0-hsc current setup
gsl 2.4 current setup
hscPipe 7.7.2 setup
ip_diffim 7.0-hsc+1 setup
ip_isr 7.1-hsc+1 setup
jointcal 7.5-hsc+1 setup
jointcal_cholmod 7.2-hsc setup
libyaml 0.1.7 current setup
lmfit 0.9.3+12 current setup
log 7.0-hsc current setup
log4cxx 0.10.0.lsst7+2 current setup
matplotlib 2.0.2-2-gce61d8e current setup
meas_algorithms 7.0-hsc+1 setup
meas_astrom 7.0-hsc+1 setup
meas_base 7.0-hsc+1 setup
meas_deblender 7.0-hsc+1 setup
meas_extensions_convolved 7.0-hsc+1 setup
meas_extensions_photometryKron 7.0-hsc+1 setup
meas_extensions_psfex 7.0-hsc+1 setup
meas_extensions_shapeHSM 7.0-hsc+1 setup
meas_modelfit 7.0-hsc+1 setup
meas_mosaic 7.0-hsc+4 setup
minuit2 5.34.14 current setup
mpi 0.0.1+3 current setup
mpi4py 2.0.0-1-g6d58a44 current setup
mpich 3.2.1 current setup
ndarray 7.0-hsc-1-g8273558 setup
numpy 1.13.1-2-g36a8a36 current setup
obs_base 7.0-hsc+1 setup
obs_subaru 7.7.2-hsc setup
obs_test 7.0-hsc+1 setup
pep8_naming 0.4.1-2-ge968bca current setup
pex_config 7.0-hsc current setup
pex_exceptions 7.0-hsc current setup
pex_policy 7.0-hsc current setup
pipe_base 7.0-hsc+1 setup
pipe_drivers 7.5-hsc+1 setup
pipe_tasks 7.0-hsc+1 setup
proxmin 7.0-hsc current setup
psfex 7.0-hsc current setup
pybind11 2.1.1-4-gd10a463 current setup
pycodestyle 2.3.1-6-g0d131b0 current setup
pyflakes 1.6.0-2-gb8a7d8b current setup
pytest 3.2.0.lsst2-5-ge2a44db current setup
pytest_cov 2.5.1+7 current setup
pytest_flake8 0.9.1-3-g14151fe current setup
pytest_forked 0.2.lsst2-3-gc5fda7b current setup
pytest_session2file 0.1.9+16 current setup
pytest_xdist 1.20.1.lsst2-3-ga186ae7 current setup
python 0.0.7-3-g85c3855 current setup
python_coverage 4.5.1+3 current setup
python_execnet 1.4.1.lsst2-3-gc0d69c6 current setup
python_future 0.16.0+6 current setup
python_mccabe 0.6.1+16 current setup
python_psutil 5.4.3+5 current setup
python_py 1.7.0+3 current setup
pyyaml 3.11.lsst2-2-ga7699fc current setup
requests 2.9.1.lsst1+7 current setup
scarlet 7.0-hsc current setup
scipy 0.0.1.lsst1-2-g5722128 current setup
scons 3.0.0.lsst1+6 current setup
sconsUtils 7.0-hsc current setup
shapelet 7.0-hsc+1 setup
skymap 7.0-hsc+1 setup
sphgeom 7.0-hsc current setup
sqlalchemy 1.2.16+3 current setup
starlink_ast 7.0-hsc current setup
utils 7.0-hsc current setup
verify 17.0.1-1-g84a5949+25 setup
verify_metrics 7.1-hsc setup
wcslib 5.13.lsst1+4 current setup
xpa 2.1.15.lsst3 current setup

(Paul Price) #11

It seems strange to me that it could be in applyLookupTable.so, when the HSC linearity correction doesn’t use a lookup table. I will investigate further tomorrow, but in the mean time perhaps you could try disabling the linearity correction (isr.doLinearize=False)?


(Ying-Tung Chen) #12

I tried the following command, then got the message of Segmentation fault again in output:

(lsst-scipipe-1172c30) [hscpipe5@ichiro 201904-pi-hp772]$ processCcd.py $WORKDIR --rerun $RERUNFRAME --calib=$WORKDIR/CALIB/ --id visit=175292 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False isr.doLinearize=False --clobber-config
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/processCcd.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/hsc/processCcd.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp772/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/calibRegistry.sqlite3
root INFO: Running: /opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.0-hsc+1/bin/processCcd.py /array/users/ytchen/processed/2019/201904-pi-hp772 --rerun JT-201904-pi-hp772 --calib=/array/users/ytchen/processed/2019/201904-pi-hp772/CALIB/ --id visit=175292 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False isr.doLinearize=False --clobber-config
processCcd INFO: Processing {‘visit’: 175292, ‘field’: ‘JTL505’, ‘dateObs’: ‘2019-04-11’, ‘pointing’: 2657, ‘filter’: ‘HSC-I2’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-11’, ‘expTime’: 24.22}
processCcd.isr INFO: Performing ISR on sensor {‘visit’: 175292, ‘field’: ‘JTL505’, ‘dateObs’: ‘2019-04-11’, ‘pointing’: 2657, ‘filter’: ‘HSC-I2’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-11’, ‘expTime’: 24.22}
processCcd.isr INFO: Converting exposure to floating point values
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7fb1c8654bd4]
/lib64/libc.so.6(+0x35990) [0x7fb1d625a990]
/lib64/libc.so.6(cfree+0x22) [0x7fb1d62ab292]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(+0x76851c) [0x7fb1cba1451c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(std::_Sp_counted_ptr_inplace<lsst::afw::detection::PeakRecord, std::allocatorlsst::afw::detection::PeakRecord, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x56) [0x7fb1cb5696f6]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_algorithms/7.0-hsc+1/lib/libmeas_algorithms.so(std::_Sp_counted_ptr_inplace<lsst::afw::detection::Footprint, std::allocatorlsst::afw::detection::Footprint, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x73) [0x7fb1cc219473]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/detection/footprint.so(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x46) [0x7fb1835aa4d6]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/detection/footprint.so(+0x1f1640) [0x7fb1835aa640]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc+1/python/lsst/ip/isr/applyLookupTable.so(+0x1f5208) [0x7fb1cc8d7208]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/7.1-hsc+1/python/lsst/ip/isr/applyLookupTable.so(+0x1f5a6e) [0x7fb1cc8d7a6e]
python(+0x10ed42) [0x559c45de6d42]
python((null)+0x48b) [0x559c45dee0ab]
python(_PyEval_EvalCodeWithName+0x48b) [0x559c45dee0ab]
python((null)+0x387) [0x559c45e50337]
python(_PyFunction_FastCallKeywords+0x387) [0x559c45e50337]
python((null)+0x14f5) [0x559c45ea95d5]
python(_PyEval_EvalFrameDefault+0x14f5) [0x559c45ea95d5]
python((null)+0xba9) [0x559c45dee7c9]
python(_PyEval_EvalCodeWithName+0xba9) [0x559c45dee7c9]
python((null)+0x387) [0x559c45e50337]
python(_PyFunction_FastCallKeywords+0x387) [0x559c45e50337]
python((null)+0x535) [0x559c45ea8615]
python(_PyEval_EvalFrameDefault+0x535) [0x559c45ea8615]
python((null)+0x2f9) [0x559c45dedf19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x559c45dedf19]
python((null)+0x3d8) [0x559c45def1e8]
python(_PyFunction_FastCallDict+0x3d8) [0x559c45def1e8]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0x5da) [0x559c45dee1fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x559c45dee1fa]
python((null)+0x3d8) [0x559c45def1e8]
python(_PyFunction_FastCallDict+0x3d8) [0x559c45def1e8]
python((null)+0x63) [0x559c45e05cb3]
python(_PyObject_Call_Prepend+0x63) [0x559c45e05cb3]
python(PyObject_Call+0x6e) [0x559c45dfa7de]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0x10b) [0x559c45deef1b]
python(_PyFunction_FastCallDict+0x10b) [0x559c45deef1b]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0x5da) [0x559c45dee1fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x559c45dee1fa]
python((null)+0x387) [0x559c45e50337]
python(_PyFunction_FastCallKeywords+0x387) [0x559c45e50337]
python((null)+0x535) [0x559c45ea8615]
python(_PyEval_EvalFrameDefault+0x535) [0x559c45ea8615]
python((null)+0x10b) [0x559c45deef1b]
python(_PyFunction_FastCallDict+0x10b) [0x559c45deef1b]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0x5da) [0x559c45dee1fa]
python(_PyEval_EvalCodeWithName+0x5da) [0x559c45dee1fa]
python((null)+0x1d4) [0x559c45deefe4]
python(_PyFunction_FastCallDict+0x1d4) [0x559c45deefe4]
python((null)+0x63) [0x559c45e05cb3]
python(_PyObject_Call_Prepend+0x63) [0x559c45e05cb3]
python(PyObject_Call+0x6e) [0x559c45dfa7de]
python((null)+0x1e3e) [0x559c45ea9f1e]
python(_PyEval_EvalFrameDefault+0x1e3e) [0x559c45ea9f1e]
python((null)+0xfb) [0x559c45e500ab]
python(_PyFunction_FastCallKeywords+0xfb) [0x559c45e500ab]
python((null)+0x535) [0x559c45ea8615]
python(_PyEval_EvalFrameDefault+0x535) [0x559c45ea8615]
python((null)+0x10b) [0x559c45deef1b]
python(_PyFunction_FastCallDict+0x10b) [0x559c45deef1b]
python((null)+0x63) [0x559c45e05cb3]
python(_PyObject_Call_Prepend+0x63) [0x559c45e05cb3]
python(+0x170cca) [0x559c45e48cca]
python((null)+0x93) [0x559c45def403]
python(_PyObject_FastCallDict+0x93) [0x559c45def403]
python(+0x14e942) [0x559c45e26942]
python(+0x1bbee4) [0x559c45e93ee4]
python((null)+0x128) [0x559c45e512d8]
python(_PyObject_FastCallKeywords+0x128) [0x559c45e512d8]
python((null)+0x4cb6) [0x559c45eacd96]
python(_PyEval_EvalFrameDefault+0x4cb6) [0x559c45eacd96]
python((null)+0xfb) [0x559c45e500ab]
python(_PyFunction_FastCallKeywords+0xfb) [0x559c45e500ab]
python((null)+0x535) [0x559c45ea8615]
python(_PyEval_EvalFrameDefault+0x535) [0x559c45ea8615]
python((null)+0x2f9) [0x559c45dedf19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x559c45dedf19]
python((null)+0x325) [0x559c45e502d5]
python(_PyFunction_FastCallKeywords+0x325) [0x559c45e502d5]
python((null)+0x4b64) [0x559c45eacc44]
python(_PyEval_EvalFrameDefault+0x4b64) [0x559c45eacc44]
python((null)+0x2f9) [0x559c45dedf19]
python(_PyEval_EvalCodeWithName+0x2f9) [0x559c45dedf19]
python(PyEval_EvalCodeEx+0x44) [0x559c45deedd4]
python(PyEval_EvalCode+0x1c) [0x559c45deedfc]
python(+0x22f9e4) [0x559c45f079e4]
python(PyRun_FileExFlags+0xa1) [0x559c45f11bd1]
python(PyRun_SimpleFileExFlags+0x1c3) [0x559c45f11dc3]
python(+0x23aedb) [0x559c45f12edb]
python((null)+0x3c) [0x559c45f12fbc]
python(_Py_UnixMain+0x3c) [0x559c45f12fbc]
/lib64/libc.so.6((null)+0xf1) [0x7fb1d6245401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7fb1d6245401]
python(+0x1dfed2) [0x559c45eb7ed2]
Segmentation fault (core dumped)


(Paul Price) #13

I’ve verified that HSC processing doesn’t go into applyLookupTable (it uses a different style of linearity correction), and that there are no relevant memory errors identified by valgrind that might cause strange behaviour. I’m therefore at a loss to explain the observed segfault. The only suggestion I can make is that you put a pdb breakpoint in lsst.ip.isr.IsrTask.run and step through to see where the segfault is happening.


(Ying-Tung Chen) #14

I don’t exactly know how to do that correctly. Where is the file of lsst.ip.isr.IsrTask.run I can edit?


(Paul Price) #15

After setting up the pipeline, do:

git clone git://github.com/HyperSuprime-Cam/ip_isr.git
cd ip_isr
setup -jr .
scons

Then edit python/lsst/ip/isr/isrTask.py, inserting at the top of the IsrTask's run method:

import pdb; pdb.set_trace()

Then run your command as usual, and you’ll be dumped into the python debugger at the appropriate place. You can step through each instruction with the n or next command or step into an instruction with the s or step command, and you can p or print the values of variables.

Please report the last code line you see at which the segfault occurs, and any other relevant observations.


(Ying-Tung Chen) #16

I am not aware of a obvious trace point in output, maybe you have an idea for that.
Few notes:
(1) That may be not relevant, I have to setup few environment parameters (OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS) to enable the running.
(2) I tested the same command in our different nodes, still got the same message.
(3) I got some warnings when I run scons (see following message)

Global pytest run completed successfully
buildConfig([“doc/doxygen.conf”], [“doc/doxygen.conf.in”])
doxygen /data/home/hscpipe5/ip_isr/doc/doxygen.conf
warning: source examples is not a readable file or directory… skipping.
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/doc/afw.tag:1: warning: Internal inconsistency: scope for class lsst::afw::table::detail::SchemaImpl::MakeItem::apply not found!
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/doc/afw.tag:1: warning: Internal inconsistency: scope for class lsst::afw::table::detail::SchemaMapperImpl::MakeKeyPair::apply not found!
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:335: warning: included file runAssembleTask.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:339: warning: included file exampleUtils.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:346: warning: included file runAssembleTask.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/isrTask.py:853: warning: Invalid list item found
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:335: warning: included file runAssembleTask.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:339: warning: included file exampleUtils.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/assembleCcdTask.py:346: warning: included file runAssembleTask.py is not found. Check your EXAMPLE_PATH
/data/home/hscpipe5/ip_isr/python/lsst/ip/isr/isrTask.py:853: warning: Invalid list item found
scons: done building targets.

Here is the last few code lines and output message:

–Call–
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/parallel.py(609)parseAndRun()
-> @classmethod
(Pdb) n
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/parallel.py(616)parseAndRun()
-> pool = startPool()
(Pdb) n
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/ctrl_pool/7.0-hsc+1/python/lsst/ctrl/pool/parallel.py(617)parseAndRun()
-> results = super(BatchParallelTask, cls).parseAndRun(*args, **kwargs)

output:

root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/singleFrameDriver.py’
root INFO: Loading config overrride file ‘/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/7.7.2-hsc/config/hsc/singleFrameDriver.py’
CameraMapper INFO: Loading exposure registry from /array/users/ytchen/processed/2019/201904-pi-hp772-core1/registry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772-core1/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /array/users/ytchen/processed/2019/201904-pi-hp772-core1/CALIB/calibRegistry.sqlite3
root INFO: Running: -c /array/users/ytchen/processed/2019/201904-pi-hp772-core1 --rerun JT-201904-pi-hp772-core1-2 --calib=/array/users/ytchen/processed/2019/201904-pi-hp772-core1/CALIB/ --id visit=174620 --config processCcd.isr.doBias=False processCcd.isr.doDark=False processCcd.isr.doFringe=False processCcd.isr.doAttachTransmissionCurve=False --clobber-config --noExit
root INFO: Processing 112 targets with a pool of 1 processes…
singleFrameDriver INFO: subaru1.asiaa.sinica.edu.tw:21140: Start processing {‘visit’: 174620, ‘field’: ‘JTL504’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
singleFrameDriver.processCcd INFO: Processing {‘visit’: 174620, ‘field’: ‘JTL504’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
singleFrameDriver.processCcd.isr INFO: Performing ISR on sensor {‘visit’: 174620, ‘field’: ‘JTL504’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
singleFrameDriver.processCcd.isr INFO: Converting exposure to floating point values
singleFrameDriver.processCcd.isr WARN: Amplifier 1 is bad.
singleFrameDriver.processCcd.isr INFO: Assembling CCD from amplifiers
singleFrameDriver.processCcd.isr INFO: Applying linearizer.
singleFrameDriver.processCcd.isr INFO: Applying crosstalk correction.
singleFrameDriver.processCcd.isr.crosstalk INFO: Applying crosstalk correction/Subaru
singleFrameDriver.processCcd.isr INFO: Widening saturation trails.
singleFrameDriver.processCcd.isr INFO: Applying brighter fatter correction.
singleFrameDriver.processCcd.isr INFO: Applying flat correction.
singleFrameDriver.processCcd.isr INFO: Setting rough magnitude zero point: 32.692803
singleFrameDriver.processCcd.isr INFO: Constructing Vignette polygon.
singleFrameDriver.processCcd.isr INFO: Adding a distortion model to the WCS.
singleFrameDriver.processCcd.isr INFO: Measuring background level:
singleFrameDriver.processCcd.isr INFO: Flattened sky level: 312.062653 +/- 10.555221
singleFrameDriver.processCcd.isr INFO: Measuring sky levels in 8x16 grids: 312.384396
singleFrameDriver.processCcd.isr INFO: Sky flatness in 8x16 grids - pp: 0.015111 rms: 0.003671
singleFrameDriver.processCcd.charImage INFO: Processing {‘visit’: 174620, ‘field’: ‘JTL504’, ‘dateObs’: ‘2019-04-10’, ‘pointing’: 2656, ‘filter’: ‘HSC-G’, ‘ccd’: 0, ‘taiObs’: ‘2019-04-10’, ‘expTime’: 30.0}
singleFrameDriver.processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
singleFrameDriver.processCcd.charImage.repair INFO: Identified 32 cosmic rays.
singleFrameDriver.processCcd.charImage.detection INFO: Detected 371 positive peaks in 35 footprints and 0 negative peaks in 0 footprints to 50 sigma
singleFrameDriver.processCcd.charImage.detection INFO: Resubtracting the background after object detection
singleFrameDriver.processCcd.charImage.measurement INFO: Measuring 35 sources (35 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/utils/7.0-hsc/lib/libutils.so(+0x14bd4) [0x7f93735dfbd4]
/lib64/libc.so.6(+0x35950) [0x7f93811f3950]
/lib64/libc.so.6(+0x16b0e0) [0x7f93813290e0]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(lsst::afw::table::AliasMap::_apply(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) const+0x64) [0x7f93670f6944]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/lib/libafw.so(lsst::afw::table::SchemaItemlsst::afw::table::Flag lsst::afw::table::Schema::findlsst::afw::table::Flag(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const+0x3c) [0x7f936714886c]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc+1/lib/libmeas_base.so(lsst::meas::base::SdssCentroidAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0xe3) [0x7f933d4b2803]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc+1/python/lsst/meas/base/sdssCentroid.so(+0x1f9aed) [0x7f931ca6baed]
/opt/hscPipe7/stack/miniconda3-4.5.12-1172c30/Linux64/meas_base/7.0-hsc+1/python/lsst/meas/base/sdssCentroid.so(+0x1fb897) [0x7f931ca6d897]
python((null)+0x264) [0x55afb23dbe24]
python(_PyMethodDef_RawFastCallKeywords+0x264) [0x55afb23dbe24]
python((null)+0x21) [0x55afb23dbf41]
python(_PyCFunction_FastCallKeywords+0x21) [0x55afb23dbf41]
python((null)+0x5266) [0x55afb2436d26]
python(_PyEval_EvalFrameDefault+0x5266) [0x55afb2436d26]
python((null)+0x10b) [0x55afb23780eb]
python(_PyFunction_FastCallDict+0x10b) [0x55afb23780eb]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x325) [0x55afb23db0f5]
python(_PyFunction_FastCallKeywords+0x325) [0x55afb23db0f5]
python((null)+0x530) [0x55afb2431ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55afb2431ff0]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x3dc) [0x55afb23783bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55afb23783bc]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x3dc) [0x55afb23783bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55afb23783bc]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x3dc) [0x55afb23783bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55afb23783bc]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x3dc) [0x55afb23783bc]
python(_PyFunction_FastCallDict+0x3dc) [0x55afb23783bc]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x14e1) [0x55afb2432fa1]
python(_PyEval_EvalFrameDefault+0x14e1) [0x55afb2432fa1]
python((null)+0x10b) [0x55afb23780eb]
python(_PyFunction_FastCallDict+0x10b) [0x55afb23780eb]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x530) [0x55afb2431ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55afb2431ff0]
python((null)+0x10b) [0x55afb23780eb]
python(_PyFunction_FastCallDict+0x10b) [0x55afb23780eb]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0xfb) [0x55afb23daecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55afb23daecb]
python((null)+0x530) [0x55afb2431ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55afb2431ff0]
python((null)+0xfb) [0x55afb23daecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55afb23daecb]
python((null)+0x4aa1) [0x55afb2436561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55afb2436561]
python((null)+0x10b) [0x55afb23780eb]
python(_PyFunction_FastCallDict+0x10b) [0x55afb23780eb]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(+0x17194a) [0x55afb23d394a]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0xab9) [0x55afb2377899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55afb2377899]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x6e6) [0x55afb24321a6]
python(_PyEval_EvalFrameDefault+0x6e6) [0x55afb24321a6]
python((null)+0xab9) [0x55afb2377899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55afb2377899]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0xab9) [0x55afb2377899]
python(_PyEval_EvalCodeWithName+0xab9) [0x55afb2377899]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python(+0x21aef7) [0x55afb247cef7]
python((null)+0x49b) [0x55afb23dc46b]
python(_PyObject_FastCallKeywords+0x49b) [0x55afb23dc46b]
python((null)+0x5266) [0x55afb2436d26]
python(_PyEval_EvalFrameDefault+0x5266) [0x55afb2436d26]
python((null)+0xfb) [0x55afb23daecb]
python(_PyFunction_FastCallKeywords+0xfb) [0x55afb23daecb]
python((null)+0x530) [0x55afb2431ff0]
python(_PyEval_EvalFrameDefault+0x530) [0x55afb2431ff0]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python((null)+0x1d4) [0x55afb23781b4]
python(_PyFunction_FastCallDict+0x1d4) [0x55afb23781b4]
python((null)+0x63) [0x55afb238f473]
python(_PyObject_Call_Prepend+0x63) [0x55afb238f473]
python(PyObject_Call+0x6e) [0x55afb238419e]
python((null)+0x1f3f) [0x55afb24339ff]
python(_PyEval_EvalFrameDefault+0x1f3f) [0x55afb24339ff]
python((null)+0x5da) [0x55afb23773ba]
python(_PyEval_EvalCodeWithName+0x5da) [0x55afb23773ba]
python((null)+0x387) [0x55afb23db157]
python(_PyFunction_FastCallKeywords+0x387) [0x55afb23db157]
python((null)+0x4aa1) [0x55afb2436561]
python(_PyEval_EvalFrameDefault+0x4aa1) [0x55afb2436561]
python((null)+0x2f9) [0x55afb23770d9]
python(_PyEval_EvalCodeWithName+0x2f9) [0x55afb23770d9]
python(PyEval_EvalCodeEx+0x44) [0x55afb2377fa4]
python(PyEval_EvalCode+0x1c) [0x55afb2377fcc]
python(+0x22f664) [0x55afb2491664]
python(PyRun_StringFlags+0x7d) [0x55afb249c77d]
python(PyRun_SimpleStringFlags+0x3f) [0x55afb249c7df]
python(+0x23a8dd) [0x55afb249c8dd]
python((null)+0x3c) [0x55afb249cc4c]
python(_Py_UnixMain+0x3c) [0x55afb249cc4c]
/lib64/libc.so.6((null)+0xf1) [0x7f93811de401]
/lib64/libc.so.6(__libc_start_main+0xf1) [0x7f93811de401]
python(+0x1df982) [0x55afb2441982]
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 21140 RUNNING AT subaru1.asiaa.sinica.edu.tw
= EXIT CODE: 139
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions


(Paul Price) #17

Whoa, that’s a different backtrace than the one you posted earlier!

What compiler did you use to build hscPipe?


(Ying-Tung Chen) #18

I ran a different frame/visit with only one core, you can see a similar message in the original post, which used 4 cores in parallel.

I am not sure what is the compiler used to build hscPipe (I had sent an email to the administrator), but ‘scons’ should use gcc 6.3.1 to build the IsrTask.

Part of output:

scons: Reading SConscript files …
EUPS integration: enabled
Checking who built the CC compiler…gcc=6.3.1
CC is gcc version 6.3.1
Checking for C++14 support
Checking whether the C++ compiler works… yes
C++14 supported with ‘-std=c++14’
Setting up environment to build package ‘ip_isr’.
Running pytest with 1 process
pytest: automated test discovery mode enabled.
scons: done reading SConscript files.
scons: Building targets …


============================= test session starts ==============================
platform linux – Python 3.7.2, pytest-3.6.2, py-1.7.0, pluggy-0.6.0
rootdir: /data/home/hscpipe5/ip_isr, inifile: setup.cfg
plugins: cov-2.5.1, session2file-0.1.9, forked-0.2, xdist-1.20.1, flake8-1.0.4, remotedata-0.3.1, openfiles-0.3.2, doctestplus-0.3.0, arraydiff-0.3


(Paul Price) #19

Same compiler we’re using at Princeton.

I think the best way to proceed from here would be to get a copy of your data, and see if I can reproduce the problem. I’ll need your calibs and the exposures it’s crashing on.


(Paul Price) #20

I just ran the data you sent me with processCcd.py and singleFrameDriver.py, and didn’t hit any segfaults. I’ll run a single CCD through with valgrind just to make sure, but I think this must be some sort of problem on your end. I suggest rebuilding the stack from scratch, and trying a different machine.