coaddDriver.py error: patch component not in range [0, 32)

Support
#1

Hi,

Using hscPipe 8.3.0, I am processing a coadd stack with a dataset covering 4 pointings of HSC fields (~7 deg2). However I encounter a error message of patch component not in range [0, 32). The experimentally test with only 3 pointings of HSC don’t show this error message. Is there a parameter to change default maximum patch number (or other solution for this error)?

Cheers,
Charles

comments:

makeDiscreteSkyMap.py  $WORKDIR  --rerun $RERUNFRAME:coadd_202010_add2 --id  visit=$VISIT2PROC_ADD2  --config skyMap.projection="TAN"

jointcal.py $WORKDIR --rerun $RERUNFRAME:coadd_202010_add2 --id visit=$VISIT2PROC_ADD2  tract=0 | tee jointcal-r2-coadd_coadd_202010_add2.log &

coaddDriver.py $WORKDIR --rerun $RERUNFRAME:coadd_202010_add2 --id tract=0 filter=HSC-R2 --selectId visit=$VISIT2PROC_ADD2 ccd=0..103 --job coadd-r-tcoadd_202010_add2 --cores=$RCHP5  -c makeCoaddTempExp.doApplySkyCorr=False makeCoaddTempExp.doApplyExternalPhotoCalib=False --clobber-config

error messages:

46264 WARN  2020-10-30T15:53:23.649+0800 coaddDriver: ichiro.asiaa.sinica.edu.tw:46264: Caught RuntimeError while detection on DataId(initialdata={'tract': 0, 'filter': 'HSC-R2', 'patch': '19,33'}, tag=set()): patch component not in range [0, 32)
46264 INFO  2020-10-30T15:53:23.661+0800 coaddDriver: ichiro.asiaa.sinica.edu.tw:46264: Traceback:
Traceback (most recent call last):
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/ctrl_pool/8.0-hsc+1/python/lsst/ctrl/pool/parallel.py", line 514, in logOperation
    yield
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_drivers/8.0-hsc+1/python/lsst/pipe/drivers/coaddDriver.py", line 322, in coadd
    idFactory = self.detectCoaddSources.makeIdFactory(patchRef)
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/8.1-hsc/python/lsst/pipe/tasks/multiBandUtils.py", line 220, in makeIdFactory
    expId = int(dataRef.get(self.config.coaddName + datasetName))
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/8.0-hsc/python/lsst/daf/persistence/butlerSubset.py", line 203, in get
    return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/8.0-hsc/python/lsst/daf/persistence/butler.py", line 1374, in get
    location = self._locate(datasetType, dataId, write=False)
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/8.0-hsc/python/lsst/daf/persistence/butler.py", line 1322, in _locate
    bypass = bypass()
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/8.0-hsc/python/lsst/daf/persistence/butler.py", line 1350, in <lambda>
    return lambda: bypassFunc(location.datasetType, pythonType, location, dataId)
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_subaru/8.3-hsc/python/lsst/obs/hsc/hscMapper.py", line 268, in bypass_deepCoaddId
    return self._computeCoaddExposureId(dataId, True)
  File "/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_subaru/8.3-hsc/python/lsst/obs/hsc/hscMapper.py", line 257, in _computeCoaddExposureId
    raise RuntimeError('patch component not in range [0, %d)' % 2**HscMapper._nbit_patch)
RuntimeError: patch component not in range [0, 32)
#2

A maximum of 32 patches in each dimension for each tract is hard-wired in the HscMapper. Your options are:

  1. Subclass HscMapper and override _nbit_patch; set your _mapper file to point to your subclass. You probably need to adjust the other _nbit_* settings so everything adds to 63.
  2. Use a larger patch size in makeDiscreteSkyMap.py (configuration option skyMap.patchInnerDimensions), which will reduce the number of patches, but produce larger images.
  3. Use a skymap more suitable for large-area surveys (e.g., rings) by using makeSkyMap.py instead of makeDiscreteSkyMap.py.
#3

Oh, I just thought of one more possibility: the rough astrometric fit done in processCcd could be diverging a bit too much, causing makeDiscreteSkyMap.py to think the surveyed area is larger than it actually is. Does the output from makeDiscreteSkyMap.py make sense and correspond to reality?

32 patches of 4000 pixels at 0.17 arcsec/pixel corresponds to just over 6 degrees. Four pointings of HSC (field width 1.5 deg) in a line would be about 6 degrees. So only if your four pointings are oriented in a line in RA or Dec with little or no overlap would you break the 32 patch limit; otherwise this astrometry problem is likely.

I’ve had unexpected results from makeDiscreteSkyMap.py before when I was accidentally including images taken of a separate field (it turned out that the observer had set the OBJECT name incorrectly). Here’s some code I used to figure out what was going on:

from lsst.daf.persistence import Butler
from lsst.geom import Box2D, SpherePoint, hours, degrees
from lsst.sphgeom import ConvexPolygon
from lsst.afw.geom import makeSkyWcs
from lsst.afw.image import bboxFromMetadata
butler = Butler("/path/to/DATA/rerun/myRerun")
dataRefList = list(butler.subset("raw", field="My Target Name", filter="HSC-R"))
target = SpherePoint((1+23/60+45/3600)*hours, (1+23/60+45/3600)*degrees)
for dataRef in dataRefList:
    if not dataRef.datasetExists("calexp"):
        continue
    md = dataRef.get("calexp_md")
    wcs = makeSkyWcs(md)
    box = Box2D(bboxFromMetadata(md))
    corners = [wcs.pixelToSky(pp).getVector() for pp in box.getCorners()]
    print(ConvexPolygon.convexHull(corners).getBoundingBox().getArea(), target.separation(wcs.pixelToSky(box.getCenter())), dataRef.dataId)
#4

Yes, my four pointings were roughly placed against Dec with the same RA, and no overlap. Actually, I am making coadd images for imageDifference.py. Trying your (2) suggestions above now, and hoping no additional issue.

Charles

#5

Be sure to check the output from makeDiscreteSkyMap.py, if you’re using it. For example, if you get something like:

makeDiscreteSkyMap INFO: tract 0 has corners (232.744, -24.283), (143.773, -24.283), (110.167, 49.248), (266.350, 49.248) (RA, Dec deg) and 914 x 914 patches

then you know there’s a problem (914 x 914 patches is far too big).

#6

Thanks. Your suggestion solved this issue, and I got the correct subtracted images. This is what I setup:

makeDiscreteSkyMap.py $WORKDIR --rerun $RERUNFRAME:coadd_202010_add3 --id visit=$VISIT2PROC_ADD2 --config skyMap.projection=“TAN” skyMap.patchInnerDimensions="(5000,5000)"

And this is the output from makeDiscreteSkyMap.py:

makeDiscreteSkyMap INFO: tract 0 has corners (28.262, 7.242), (21.927, 7.242), (21.863, 13.526), (28.326, 13.526) (RA, Dec deg) and 27 x 27 patches

1 Like