Reducing Suprime-cam Data

price · April 6, 2018, 2:02am

So what’s in the calibration registry? Please post the results of running:

sqlite3 /home/lsst/DATA/CALIB/calibRegistry.sqlite3
.header on
SELECT * FROM bias;

If that’s empty, please post the ingestCalibs.py command you used.

kpittman · April 6, 2018, 4:29pm

bash-4.2$ sqlite3 /home/lsst/DATA/CALIB/calibRegistry.sqlite3
SQLite version 3.13.0 2016-05-18 10:57:30
Enter ".help" for usage hints.
sqlite> .header on
sqlite> SELECT * FROM bias;
id|filter|ccd|calibDate|validStart|validEnd
1|NONE|0|2009-04-21|2006-07-26|2012-01-16
2|NONE|1|2009-04-21|2006-07-26|2012-01-16
3|NONE|2|2009-04-21|2006-07-26|2012-01-16
4|NONE|3|2009-04-21|2006-07-26|2012-01-16
5|NONE|4|2009-04-21|2006-07-26|2012-01-16
6|NONE|5|2009-04-21|2006-07-26|2012-01-16
7|NONE|6|2009-04-21|2006-07-26|2012-01-16
8|NONE|7|2009-04-21|2006-07-26|2012-01-16
9|NONE|8|2009-04-21|2006-07-26|2012-01-16
10|NONE|9|2009-04-21|2006-07-26|2012-01-16

The ingestCalibs.py command I ran was:

ingestCalibs.py DATA --rerun n5846 --calib /home/lsst/DATA/CALIB --validity 1000 /home/lsst/DATA/rerun/n5846/BIAS/2009-04-21/NONE/BIAS-*.fits

price · April 6, 2018, 4:43pm

Oh, maybe it’s not related to the calib registry, but pointing to it. Please try:

processCcd.py DATA --calib DATA/CALIB --rerun processCcdOutput --id field=N5846_1

kpittman · April 6, 2018, 5:06pm

That gives me:

bash-4.2$ processCcd.py DATA --calib DATA/CALIB --rerun processCcdOutput --id field=N5846_1
root INFO: Loading config overrride file '/home/lsst/obs_subaru/config/processCcd.py'
root INFO: Loading config overrride file '/home/lsst/obs_subaru/config/suprimecam/processCcd.py'
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/bin/processCcd.py", line 25, in <module>
    ProcessCcdTask.parseAndRun()
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/cmdLineTask.py", line 581, in parseAndRun
    parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/argumentParser.py", line 614, in parse_args
    namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/butler.py", line 521, in __init__
    self._getCfgs(repoDataList)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/butler.py", line 821, in _getCfgs
    cfg, repoData.repoArgs))
RuntimeError: The RepositoryArgs and RepositoryCfg must match for writable repositories, RepositoryCfg:RepositoryCfg(root='/home/lsst/DATA/rerun/processCcdOutput', mapper=<class 'lsst.obs.suprimecam.suprimecamMapper.SuprimecamMapper'>, mapperArgs={}, parents=[RepositoryCfg(root='/home/lsst/DATA', mapper=<class 'lsst.obs.suprimecam.suprimecamMapper.SuprimecamMapper'>, mapperArgs={}, parents=[], policy=None)], policy=None), RepositoryArgs:RepositoryArgs(root='/home/lsst/DATA/rerun/processCcdOutput', cfgRoot=None, mapper=None, mapperArgs={'calibRoot': '/home/lsst/DATA/CALIB'}, tags=set(), mode='rw', policy=None)

price · April 6, 2018, 6:17pm

That’s because the rerun was created with the old (presumably wrong) configuration. Kill it and start over.

rm -r DATA/rerun/processCcdOutput

kpittman · April 9, 2018, 5:30pm

That worked. Now I am getting:

processCcd FATAL: Failed on dataId={'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 0, 'taiObs': '2009-04-21', 'expTime': 460.0}: ColortermNotFoundError: No colorterm dict found with photoCatName 'ps1_pv3_3pi_20170110'
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
    result = task.run(dataRef, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/processCcd.py", line 199, in run
    icSourceCat=charRes.sourceCat,
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/calibrate.py", line 433, in run
    icSourceCat=icSourceCat,
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/calibrate.py", line 536, in calibrate
    photoRes = self.photoCal.run(exposure, sourceCat=sourceCat, expId=exposureIdInfo.expId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/photoCal.py", line 471, in run
    arrays = self.extractMagArrays(matches=matches, filterName=filterName, sourceKeys=sourceKeys)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/photoCal.py", line 326, in extractMagArrays
    filterName=filterName, photoCatName=self.config.photoCatName, doRaise=True)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/colorterms.py", line 183, in getColorterm
    "No colorterm dict found with photoCatName %r" % photoCatName)
lsst.pipe.tasks.colorterms.ColortermNotFoundError: No colorterm dict found with photoCatName 'ps1_pv3_3pi_20170110'

I have added the code from this post to my config/processCcd.py (both the code under LSST-style format and common configuration). More specifically, I added:

config.calibrate.astromRefObjLoader.retarget(LoadIndexedReferenceObjectsTask)
config.calibrate.astromRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110"
config.calibrate.photoRefObjLoader.retarget(LoadIndexedReferenceObjectsTask)
config.calibrate.photoRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110"
config.calibrate.photoCal.photoCatName = "ps1_pv3_3pi_20170110"

and

colorterms = config.calibrate.photoCal.colorterms
from lsst.pipe.tasks.colorterms import ColortermDict, Colorterm
colorterms.data["ps1*"] = ColortermDict(data={
       'g': Colorterm(primary="g", secondary="r", c0=-0.00928, c1=-0.0824),
       'r': Colorterm(primary="r", secondary="i", c0=-0.00282, c1=-0.0498, c2=-0.0149),
       'i': Colorterm(primary="i", secondary="z", c0=0.00186, c1=-0.140, c2=-0.0196),
       'z': Colorterm(primary="z", secondary="i", c0=-4.03e-4, c1=0.0967, c2=0.0210),
       'B': Colorterm(primary="g", secondary="r", c0=0.02461907, c1=0.20098328, c2=0.00858468),
       'V': Colorterm(primary="g", secondary="r", c0=-0.03117934, c1=-0.63134136, c2=0.05056544),
       'R': Colorterm(primary="r", secondary="i", c0=-0.01179613, c1=-0.25403307, c2=0.00696479),
       'I': Colorterm(primary="i", secondary="r", c0=0.01078282, c1=0.26727768, c2=0.00747123),
})

the latter coming from obs_subaru/config/suprimecam/colorterms.py.

price · April 9, 2018, 6:00pm

The color terms in obs_subaru/config/suprimecam/colorterms.py are for the SDSS catalog, not PS1.

I think you want to set calibrate.photoCal.applyColorTerms=False to disable the application of color terms.

kpittman · April 11, 2018, 6:57pm

I am now running into a segmentation fault in processCcd.py. The full error is (apologies for the giant block of text):

processCcd INFO: Processing {'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 8, 'taiObs': '2009-04-21', 'expTime': 460.0}
processCcd.isr INFO: Performing ISR on sensor {'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 8, 'taiObs': '2009-04-21', 'expTime': 460.0}
daf.persistence.butler INFO: NoResults or IOError (continuing search) while evaluating bypass function for Dataset type:raw_md Data ID:DataId(initialdata={'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 8, 'taiObs': '2009-04-21', 'expTime': 460.0}, tag=set()) at location lsst.afw.image.DecoratedImageU at FitsStorage(N5846_1/2009-04-21/01937/W-S-G+/SUPA01090458.fits)
processCcd.isr.crosstalk INFO: Applying crosstalk corrections to CCD 8 based on Yagi+2012
processCcd.isr INFO: Set 0 BAD pixels to 1855.65
processCcd.isr INFO: Adding a distortion model to the WCS
processCcd.isr INFO: Flattened sky level: 1855.938904 +/- 28.985080
processCcd.isr INFO: Measuring sky levels in 8x16 grids: 1854.532196
processCcd.isr INFO: Sky flatness in 8x16 grids - pp: 0.028817 rms: 0.005004
processCcd.isr INFO: Setting rough magnitude zero point: 35.656895
processCcd.charImage INFO: Processing {'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 8, 'taiObs': '2009-04-21', 'expTime': 460.0}
processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processCcd.charImage.repair INFO: Identified 1205 cosmic rays.
processCcd.charImage.detection INFO: Detected 1436 positive peaks in 345 footprints and 12 negative peaks in 4 footprints to 50 sigma
processCcd.charImage.detection INFO: Resubtracting the background after object detection
processCcd.charImage.measurement INFO: Measuring 345 sources (345 parents, 0 children)
Caught signal 11, backtrace follows:
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/utils/14.0-10-gaad1ee2+1/lib/libutils.so(+0x15214) [0x7f8d8288f214]
/lib64/libc.so.6(+0x35270) [0x7f8d8bca7270]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/lib/libmeas_modelfit.so(+0xefb08) [0x7f8d1e6f1b08]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/lib/libmeas_modelfit.so(Eigen::internal::general_matrix_matrix_triangular_product<long, double, 1, false, double, 0, false, 0, 2, 0>::run(long, long, double const*, long, double const*, long, double*, long, double const&)+0x430) [0x7f8d1e6f4920]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/lib/libmeas_modelfit.so(lsst::meas::modelfit::Optimizer::_computeDerivatives()+0x1742) [0x7f8d1e6ea5b2]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/lib/libmeas_modelfit.so(lsst::meas::modelfit::Optimizer::_stepImpl(int, lsst::meas::modelfit::OptimizerHistoryRecorder const*, lsst::afw::table::CatalogT<lsst::afw::table::BaseRecord>*)+0x1217) [0x7f8d1e6ef8b7]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/lib/libmeas_modelfit.so(lsst::meas::modelfit::Optimizer::_runImpl(lsst::meas::modelfit::OptimizerHistoryRecorder const*, lsst::afw::table::CatalogT<lsst::afw::table::BaseRecord>*)+0x111) [0x7f8d1e6f06e1]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/lib/libmeas_modelfit.so(+0x53afa) [0x7f8d1e655afa]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/lib/libmeas_modelfit.so(lsst::meas::modelfit::CModelAlgorithm::_applyImpl(lsst::meas::modelfit::CModelResult&, lsst::afw::image::Exposure<float, int, float> const&, lsst::shapelet::MultiShapeletFunction const&, lsst::afw::geom::Point<double, 2> const&, lsst::afw::geom::ellipses::Quadrupole const&, double, double, int) const+0x888) [0x7f8d1e65f648]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/lib/libmeas_modelfit.so(lsst::meas::modelfit::CModelAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0x3fb) [0x7f8d1e6602db]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/python/lsst/meas/modelfit/cmodel/cmodel.so(void pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, lsst::meas::modelfit::CModelAlgorithm, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg>(void (lsst::meas::modelfit::CModelAlgorithm::*)(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const, pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg const&)::{lambda(lsst::meas::modelfit::CModelAlgorithm const*, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&)#1}, void, lsst::meas::modelfit::CModelAlgorithm const*, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg>(pybind11::cpp_function::initialize<void, lsst::meas::modelfit::CModelAlgorithm, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg>(void (lsst::meas::modelfit::CModelAlgorithm::*)(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const, pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg const&)::{lambda(lsst::meas::modelfit::CModelAlgorithm const*, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&)#1}&&, void (*)(lsst::meas::modelfit::CModelAlgorithm const*, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg const&)::{lambda(pybind11::detail::function_call&)#3}::operator()(pybind11::detail::function_call) const+0xe2) [0x7f8d166e9ad2]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/python/lsst/meas/modelfit/cmodel/cmodel.so(void pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, lsst::meas::modelfit::CModelAlgorithm, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg>(void (lsst::meas::modelfit::CModelAlgorithm::*)(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const, pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg const&)::{lambda(lsst::meas::modelfit::CModelAlgorithm const*, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&)#1}, void, lsst::meas::modelfit::CModelAlgorithm const*, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg>(pybind11::cpp_function::initialize<void, lsst::meas::modelfit::CModelAlgorithm, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg>(void (lsst::meas::modelfit::CModelAlgorithm::*)(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const, pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg const&)::{lambda(lsst::meas::modelfit::CModelAlgorithm const*, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&)#1}&&, void (*)(lsst::meas::modelfit::CModelAlgorithm const*, lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call)+0xe) [0x7f8d166e9c8e]
/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_modelfit/14.0-3-gda7b0c7+14/python/lsst/meas/modelfit/cmodel/cmodel.so(pybind11::cpp_function::dispatcher(_object*, _object*, _object*)+0xdc6) [0x7f8d166ee0e6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x102) [0x7f8d8cc25302]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x102) [0x7f8d8cc25302]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x14895b) [0x7f8d8ccaa95b]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x146514) [0x7f8d8cca8514]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x285) [0x7f8d8cca9515]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x285) [0x7f8d8cca9515]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x166) [0x7f8d8cbcdce6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x166) [0x7f8d8cbcdce6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0xcc) [0x7f8d8cbcdf3c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f8d8cbcdf3c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x10c) [0x7f8d8cca939c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x10c) [0x7f8d8cca939c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x166) [0x7f8d8cbcdce6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x166) [0x7f8d8cbcdce6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0xcc) [0x7f8d8cbcdf3c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f8d8cbcdf3c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x148b2a) [0x7f8d8ccaab2a]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x148b2a) [0x7f8d8ccaab2a]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f8d8cca9583]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x9f1a1) [0x7f8d8cc011a1]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x148b2a) [0x7f8d8ccaab2a]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f8d8cca9583]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x9f1a1) [0x7f8d8cc011a1]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x148b2a) [0x7f8d8ccaab2a]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f8d8cca9583]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x9f1a1) [0x7f8d8cc011a1]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x148b2a) [0x7f8d8ccaab2a]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f8d8cca9583]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x9f1a1) [0x7f8d8cc011a1]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x148b2a) [0x7f8d8ccaab2a]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x31cc) [0x7f8d8ccae2cc]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f8d8cca9583]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x9f1a1) [0x7f8d8cc011a1]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x10c) [0x7f8d8cca939c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x10c) [0x7f8d8cca939c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x166) [0x7f8d8cbcdce6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x166) [0x7f8d8cbcdce6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0xcc) [0x7f8d8cbcdf3c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f8d8cbcdf3c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ec9) [0x7f8d8ccaefc9]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x146514) [0x7f8d8cca8514]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x285) [0x7f8d8cca9515]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x285) [0x7f8d8cca9515]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x166) [0x7f8d8cbcdce6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x166) [0x7f8d8cbcdce6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0xcc) [0x7f8d8cbcdf3c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcc) [0x7f8d8cbcdf3c]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyObject_Call+0x56) [0x7f8d8cbcdfd6]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0xe444f) [0x7f8d8cc4644f]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x9e) [0x7f8d8cbcdc1e]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x9e) [0x7f8d8cbcdc1e]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x142249) [0x7f8d8cca4249]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0xa520e) [0x7f8d8cc0720e]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0xa5668) [0x7f8d8cc07668]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0xda2ac) [0x7f8d8cc3c2ac]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x9e) [0x7f8d8cbcdc1e]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x9e) [0x7f8d8cbcdc1e]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x14895b) [0x7f8d8ccaa95b]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x146514) [0x7f8d8cca8514]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x148c88) [0x7f8d8ccaac88]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x148b2a) [0x7f8d8ccaab2a]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0((null)+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x2c40) [0x7f8d8ccadd40]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7f8d8cca9100]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7f8d8cca9583]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCode+0x3b) [0x7f8d8cca95cb]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyRun_FileExFlags+0x130) [0x7f8d8ccdbee0]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(PyRun_SimpleFileExFlags+0x173) [0x7f8d8ccdd4a3]
/opt/lsst/software/stack/python/miniconda3-4.3.21/bin/../lib/libpython3.6m.so.1.0(Py_Main+0xe25) [0x7f8d8ccf88d5]
python(main+0x16d) [0x400c1d]
/lib64/libc.so.6((null)+0xf5) [0x7f8d8bc93c05]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8d8bc93c05]
python() [0x4009e9]
Segmentation fault

what’s particularly confusing for me is that the step it crashed on ran fine for 8 prior images, specifically 'ccd': 0-7. I found forum posts stating that most segfaults are due to a mismatch of library versions, so I verified my versions (obs_subaru DM-13148, LSST stack 7-stack-lsst_distrib-w_2018_11).

price · April 11, 2018, 7:12pm

Most often segfaults are due to mismatched versions, but there are other possibilities. What setup commands have you run, and what packages have you built personally?
Can you run processCcd.py under gdb (e.g., gdb --args $(which python) $(which processCcd.py) ...) and get a backtrace?

kpittman · April 17, 2018, 5:12pm

I’m not sure what happened, but in the process of making a new docker container with the correct permissions to run gdb, the segfault was fixed and processCcd.py ran all the way through. I assume starting over fixed any library mismatches.

kpittman · May 1, 2018, 12:56am

Hello again,
I was hoping to clerify the next steps to coadd and mosaic the Suprime-cam data. This tutorial just runs makeDiscreteSkyMap.py and makeCoaddTempExp.py, but I’ve seen coaddDriver.py mentioned in relation to the other two and want to double check. Also, when I ran makeDiscreteSkyMap.py with
makeDiscreteSkyMap.py DATA --id field=N5846_1 --rerun processCcdOutput:coadd --config skyMap.projection="TAN" and got

daf.persistence.butler INFO: NoResults or IOError (continuing search) while evaluating bypass function for Dataset type:calexp_md Data ID:DataId(initialdata={'field': 'N5846_1', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109076, 'ccd': 9, 'dateObs': '2009-04-21', 'taiObs': '2009-04-21', 'expTime': 460.0}, tag=set()) at location lsst.afw.image.ExposureF at FitsStorage(01937/W-S-G+/corr/CORR01090769.fits)

for each image, and I wasn’t sure if this was expected behavior, though it did run. I also experienced this when running makeCoaddTempExp.py, though it did not finish, and stoped with the message ‘Killed’. I ran makeCoaddTempExp.py with
makeCoaddTempExp.py DATA --rerun coadd --selectId filter=W-S-G+ --id filter=W-S-G+ tract=0 patch=0,0^0,1^0,2^0,3^1,0^1,1^1,2^1,3^2,0^2,1^2,2^2,3^3,0^3,1^3,2^3,3 --config doApplyUberCal=False doApplySkyCorr=False
Thank you!

price · May 1, 2018, 2:09am

coaddDriver.py combines makeCoaddTempExp.py, assembleCoadd.py and detectCoaddSources.py in a form where it can be run over multiple nodes (in the same way as singleFrameDriver.py). Do be warned, however, that spreading it too wide (I typically use a maximum of about 20 cores) can overwhelm some cluster filesystems due to the multitude of small reads in the early stage.

The NoResults or IOError is not something you should worry about greatly. It probably means that a CCD didn’t pass successfully through a previous stage.

You may want to run meas_mosaic (mosaic.py) on your data, but I’m not sure if it still works on Suprime-Cam. Here’s an example command-line from HSC that you can adapt:

mosaic.py /tigress/HSC/HSC/ --rerun price/ikeda-20180430 --calib /tigress/pprice/ikeda-20180410/CALIB_s18a --id tract=8766 ccd=0..8^10..103 visit=84654^84656^84658^84662^84664^84836^137270^137272^137274^137276^137278^138618

I’ve had reports from others about processes failing with the Killed message, but I have not recently experienced the problem myself. I believe this is the work of the system’s out-of-memory assassin, but I can’t think what might be causing it. I would be grateful to anyone who could help me reproduce the problem.

kpittman · May 8, 2018, 9:49pm

I’ve been playing around with makeCoaddTempExp.py and I noticed that it crashes and gives the Killed message at the same spot, no matter how many patches I try to coadd. If I run
makeCoaddTempExp.py DATA --rerun coadd --selectId filter=W-S-G+ --id filter=W-S-G+ tract=0 patch=0,0^0,1^0,2^0,3^1,0^1,1^1,2^1,3^2,0^2,1^2,2^2,3^3,0^3,1^3,2^3,3 --config doApplyUberCal=False doApplySkyCorr=False
or
makeCoaddTempExp.py DATA --rerun coadd --selectId filter=W-S-G+ --id filter=W-S-G+ tract=0 patch=0,3 --config doApplyUberCal=False doApplySkyCorr=False
it will crash at the same spot, trying to coadd patch 0,3.

makeCoaddTempExp INFO: Selected 10 calexps for patch DataId(initialdata={'filter': 'W-S-G+', 'tract': 0, 'patch': '0,3'}, tag=set())
makeCoaddTempExp INFO: Processing 10 existing calexps for patch DataId(initialdata={'filter': 'W-S-G+', 'tract': 0, 'patch': '0,3'}, tag=set())
makeCoaddTempExp INFO: Processing 9 warp exposures for patch DataId(initialdata={'filter': 'W-S-G+', 'tract': 0, 'patch': '0,3'}, tag=set())
makeCoaddTempExp INFO: Processing Warp 0/9: id=DataId(initialdata={'visit': 109045, 'filter': 'W-S-G+', 'patch': '0,3', 'tract': 0}, tag=set())
makeCoaddTempExp INFO: Processing calexp 1 of 1 for this Warp: id={'filter': 'W-S-G+', 'pointing': 1937, 'visit': 109045, 'ccd': 6, 'field': 'N5846_1', 'dateObs': '2009-04-21', 'taiObs': '2009-04-21', 'expTime': 460.0}
makeCoaddTempExp.warpAndPsfMatch.psfMatch INFO: compute Psf-matching kernel
makeCoaddTempExp.warpAndPsfMatch.psfMatch INFO: Adjusted dimensions of reference PSF model from (23, 23) to (341, 341)
Killed

I’m wondering if the Adjusted dimensions of reference PSF model from (23, 23) to (341, 341) has something to do with it, as that’s where it is crashing each time and the other patches only adjusted from 23 to ~50-70 instead of 341. Do you have any suggestions on how I can track down the issue? Or is the answer to reduce the data on something with more available memory? Thanks

price · May 8, 2018, 11:24pm

Can you try deactivating the PSF matching (doPsfMatch=False) to see if that allows it to complete?

kpittman · May 9, 2018, 2:19am

Unfortunately, deactivating the PST matching did not work. I have found that it is patches 0,3 ^ 1,3 ^ 2,3 ^ 3,3 which are not able to run.

Edit: I just decided to run on a machine with more available memory and it worked. I really appreciate the continued help with this!

price · May 10, 2018, 1:51pm

How much memory per core did you have before, and how much now? And how many jobs are you running in parallel (-j)?

kpittman · May 10, 2018, 4:28pm

Initially I was using a Docker container with only 2GB of allocated memory, and I am now using 8GB. This is still not enough for a couple of the patches, so I will be switching to something with more. I have not been using the -j option, so I guess whatever the default is.

price · May 10, 2018, 4:42pm

Ah! 2 GB is much less than I would recommend. You want at least 4 GB per core, and sometimes (depending on the source density), even twice that amount is sometimes too small.

kpittman · June 28, 2018, 4:15pm

Hello,
I recently started working on this again after a short break, and I was wondering if there is a task or function in the pipeline that will mosaic the coadded patches made by makeCoaddTempExp.py and assembleCoadd.py into a final image, or would that be something I have to do with other software? I’ve been looking around but haven’t found a lot of information.
Thank you!

price · July 6, 2018, 2:06pm

(Sorry for the delay; I was on vacation.)

I’m not aware of anything formal in our stack that will mosaic the coadded patches, as we have focussed on reducing survey data where the full mosaic would be impractical to handle. There may be some informal solutions invented by various people, as the code isn’t difficult.

An alternative solution would be to use a different skymap, so that the tract of interest contains a single patch: see the patchInnerDimensions configuration parameter.