Hi all,
I’m trying to generate some sky-frames for archival HSC data using the release 21 LSST stack, using the following command:
constructSky.py . --calib=CALIB/ --rerun globalsky --job sky --batch-type=smp --cores=128 --id filter=‘HSC-R’ field=M81_1^M81_2
However, for one of the filters, execution abruptly stops with a TaskError:
…
60186 INFO 2021-02-01T09:59:57.482-0600 sky.isr: Converting exposure to floating point values.
60155 WARN 2021-02-01T09:59:57.483-0600 sky.isr: Amplifier 0 is bad.
60251 INFO 2021-02-01T09:59:57.483-0600 sky.isr: Setting rough magnitude zero point: 35.192803
60251 INFO 2021-02-01T09:59:57.483-0600 sky.isr: Measuring background level.
/opt/lsst/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-cb4e2dc/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3373: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
/opt/lsst/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-cb4e2dc/lib/python3.7/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
/opt/lsst/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-cb4e2dc/lib/python3.7/site-packages/numpy/core/_methods.py:234: RuntimeWarning: Degrees of freedom <= 0 for slice
keepdims=keepdims)
/opt/lsst/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-cb4e2dc/lib/python3.7/site-packages/numpy/core/_methods.py:195: RuntimeWarning: invalid value encountered in true_divide
arrmean, rcount, out=arrmean, casting=‘unsafe’, subok=False)
/opt/lsst/conda/miniconda3-py37_4.8.2/envs/lsst-scipipe-cb4e2dc/lib/python3.7/site-packages/numpy/core/_methods.py:226: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
TaskError on anacreon:60263 in run: All pixels masked. Cannot estimate background
Traceback (most recent call last):
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py”, line 112, in wrapper
return func(*args, **kwargs)
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py”, line 1069, in run
while not menucommand:
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py”, line 239, in wrapper
return func(*args, **kwargs)
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py”, line 1087, in reduce
result = self._processQueue(context, func, [(index, data)], *args, **kwargs)[0]
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py”, line 546, in _processQueue
return self._reduceQueue(context, None, func, queue, *args, **kwargs)
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py”, line 572, in _reduceQueue
resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ctrl_pool/21.0.0+7927753e06/python/lsst/ctrl/pool/pool.py”, line 572, in
resultList = [func(self._getCache(context, i), data, *args, **kwargs) for i, data in queue]
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/21.0.0+c33e90402a/python/lsst/pipe/drivers/constructCalibs.py”, line 1259, in measureBackground
exposure = self.processSingleBackground(dataRef)
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/21.0.0+c33e90402a/python/lsst/pipe/drivers/constructCalibs.py”, line 1284, in processSingleBackground
self.maskObjects.run(exposure, self.config.mask)
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/21.0.0+c33e90402a/python/lsst/pipe/drivers/background.py”, line 802, in run
self.findObjects(exposure)
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_drivers/21.0.0+c33e90402a/python/lsst/pipe/drivers/background.py”, line 816, in findObjects
bg = self.subtractBackground.run(exposure).background
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/meas_algorithms/21.0.0+23efe4bbff/python/lsst/meas/algorithms/subtractBackground.py”, line 239, in run
fitBg = self.fitBackground(maskedImage)
File “/opt/lsst/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/meas_algorithms/21.0.0+23efe4bbff/python/lsst/meas/algorithms/subtractBackground.py”, line 326, in fitBackground
raise pipeBase.TaskError(“All pixels masked. Cannot estimate background”)
lsst.pipe.base.task.TaskError: All pixels masked. Cannot estimate background
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 116
The question now is how to get around this to move forward.
Is there a way to find which exact exposure is causing this so I can exclude it from generating the sky? I suspect the initial number in the log is the process-ID, rather than the visit number?
Or even better some workaround to simply ignore CCDs for which background estimation fails (on a related note: does this prevent the actual science frames from being processed as well?).
Ultimately I suspect that the large galaxy M81 is causing this, and while M81 isn’t the actual target I’m after it would be nice to have it in the final data product.
Any help will be much appreciated.
Thanks
Ralf