Failure finding flux fields in processEimages.py

rbliu · December 7, 2017, 5:36pm

Hi!
We are trying to ingest and do some astrometry calibration on a PhoSim eimage with ImageProcessingPipelines.

We took (ra, dec) = (1.45, -0.89) as a test, and it was running smoothly for
$ generate_dc_reference_catalog.py 1.45 -0.89 2.0,
$ ingestReferenceCatalog.py repo/ ref_cat.txt and
$ ingestSimImages.py repo/ /path/to/lsst_e_138143_f5_R22_S11_E000.fits.gz --mode link

But a RuntimeError showed up when running
$ processEimage.py repo/ --output repo/ --id visit=138143 --configfile configs/processEimage.py

It seems failed to find two flux fields in the ingested data. Let me paste the whole traceback below:
(we are using w_2017_26 of DM stack and a freshly cloned ImageProcessingPipelines on NERSC.)

root INFO: Loading config overrride file '/global/cscratch1/sd/jchiang8/L2_demo/obs_lsstSim/config/processEimage.py'

root INFO: Running: /global/cscratch1/sd/jchiang8/L2_demo/obs_lsstSim/bin/processEimage.py repo/ --output repo/ --id visit=138143 --configfile configs/processEimage.py --clobber-config
fatal: index file open failed: Permission denied
processEimage INFO: Processing {'raft': '2,2', 'visit': 138143, 'sensor': '1,1', 'filter': 'y'}
CameraMapper WARN: Key="ROTANG" not in medata
processEimage.charImage INFO: Processing {'raft': '2,2', 'visit': 138143, 'sensor': '1,1', 'filter': 'y'}
processEimage.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processEimage.charImage.repair INFO: Identified 5652 cosmic rays.
processEimage.charImage.detection INFO: Detected 826 positive sources to 50 sigma.
processEimage.charImage.detection INFO: Resubtracting the background after object detection
processEimage.charImage.measurement INFO: Measuring 826 sources (826 parents, 0 children) 
processEimage.charImage.measurePsf INFO: Measuring PSF
/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_algorithms/13.0-14-gb26cf08/python/lsst/meas/algorithms/objectSizeStarSelector.py:400: RuntimeWarning: invalid value encountered in less
  bad = numpy.logical_or(bad, width < self.config.widthMin)
/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_algorithms/13.0-14-gb26cf08/python/lsst/meas/algorithms/objectSizeStarSelector.py:401: RuntimeWarning: invalid value encountered in greater
  bad = numpy.logical_or(bad, width > self.config.widthMax)
processEimage.charImage.measurePsf INFO: PSF star selector found 624 candidates
processEimage.charImage.measurePsf.psfDeterminer WARN: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
processEimage.charImage.measurePsf INFO: PSF determination using 492/624 stars.
processEimage.charImage INFO: iter 1; PSF sigma=1.74, dimensions=(41, 41); median background=2139.98
processEimage.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processEimage.charImage.repair INFO: Identified 8049 cosmic rays.
processEimage.charImage.detection INFO: Detected 830 positive sources to 50 sigma.
processEimage.charImage.detection INFO: Resubtracting the background after object detection
processEimage.charImage.measurement INFO: Measuring 830 sources (830 parents, 0 children) 
processEimage.charImage.measurePsf INFO: Measuring PSF
/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_algorithms/13.0-14-gb26cf08/python/lsst/meas/algorithms/objectSizeStarSelector.py:400: RuntimeWarning: invalid value encountered in less
  bad = numpy.logical_or(bad, width < self.config.widthMin)
/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_algorithms/13.0-14-gb26cf08/python/lsst/meas/algorithms/objectSizeStarSelector.py:401: RuntimeWarning: invalid value encountered in greater
  bad = numpy.logical_or(bad, width > self.config.widthMax)
processEimage.charImage.measurePsf INFO: PSF star selector found 439 candidates
processEimage.charImage.measurePsf.psfDeterminer WARN: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value
processEimage.charImage.measurePsf INFO: PSF determination using 318/439 stars.
processEimage.charImage INFO: iter 2; PSF sigma=1.73, dimensions=(41, 41); median background=2139.92
processEimage.charImage.repair INFO: Identified 8215 cosmic rays.
processEimage.charImage.measurement INFO: Measuring 830 sources (830 parents, 0 children) 
processEimage.charImage.measureApCorr INFO: Measuring aperture corrections for 3 flux fields
processEimage.charImage.measureApCorr INFO: Aperture correction for base_PsfFlux: RMS 0.072091 from 298
processEimage.charImage.measureApCorr INFO: Aperture correction for base_GaussianFlux: RMS 0.071558 from 285
processEimage.charImage.measureApCorr INFO: Aperture correction for ext_photometryKron_KronFlux: RMS 0.043489 from 291
processEimage.charImage.applyApCorr INFO: Applying aperture corrections to 3 flux fields
processEimage.calibrate INFO: Processing {'raft': '2,2', 'visit': 138143, 'sensor': '1,1', 'filter': 'y'}
processEimage.calibrate.detection INFO: Detected 3420 positive sources to 5 sigma.
processEimage.calibrate.detection INFO: Resubtracting the background after object detection
processEimage.calibrate.deblend INFO: Deblending 3420 sources
processEimage.calibrate.deblend INFO: Deblended: of 3420 sources, 362 were deblended, creating 828 children, total 4248 sources
processEimage.calibrate.measurement INFO: Measuring 4248 sources (3420 parents, 828 children) 
processEimage.calibrate.applyApCorr INFO: Applying aperture corrections to 3 flux fields
processEimage.calibrate INFO: Copying flags from icSourceCat to sourceCat for 716 sources
processEimage.calibrate.astromRefObjLoader INFO: Loading reference objects using center (1999.5, 2035.5) pix = IcrsCoord(1.4494221, -0.8855439) sky and radius 0.18605064076156072 deg
processEimage FATAL: Failed on dataId={'raft': '2,2', 'visit': 138143, 'sensor': '1,1', 'filter': 'y'}: Could not find flux field(s) y_camFlux, y_flux
Traceback (most recent call last):
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/cmdLineTask.py", line 376, in __call__
    result = task.run(dataRef, **kwargs)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-38-gf73ba12/python/lsst/pipe/tasks/processCcd.py", line 198, in run
    icSourceCat=charRes.sourceCat,
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-38-gf73ba12/python/lsst/pipe/tasks/calibrate.py", line 439, in run
    icSourceCat=icSourceCat,
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-38-gf73ba12/python/lsst/pipe/tasks/calibrate.py", line 530, in calibrate
    sourceCat=sourceCat,
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_astrom/13.0-14-g9415442+3/python/lsst/meas/astrom/astrometry.py", line 190, in run
    res = self.solve(exposure=exposure, sourceCat=sourceCat)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_astrom/13.0-14-g9415442+3/python/lsst/meas/astrom/astrometry.py", line 220, in solve
    calib=expMd.calib,
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_algorithms/13.0-14-gb26cf08/python/lsst/meas/algorithms/loadReferenceObjects.py", line 215, in loadPixelBox
    loadRes = self.loadSkyCircle(ctrCoord, maxRadius, filterName)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+5/python/lsst/pipe/base/timer.py", line 121, in wrapper
    res = func(self, *args, **keyArgs)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_algorithms/13.0-14-gb26cf08/python/lsst/meas/algorithms/loadIndexedReferenceObjects.py", line 78, in loadSkyCircle
    fluxField = getRefFluxField(schema=refCat.schema, filterName=filterName)
  File "/global/common/cori/contrib/lsst/lsstDM/w_2017_26_py3/stack/miniconda3-4.2.12-7c8e67/Linux64/meas_algorithms/13.0-14-gb26cf08/python/lsst/meas/algorithms/loadReferenceObjects.py", line 65, in getRefFluxField
    raise RuntimeError("Could not find flux field(s) %s" % (", ".join(fluxFieldList)))
RuntimeError: Could not find flux field(s) y_camFlux, y_flux

merlin · December 7, 2017, 7:04pm

My guess is that the names of the filters are different in the PhoSim images and the reference catalog you’re using.

You can create a mapping between the two as shown below. This example snippet shows how to do this for the three different stages at which this might be relevant, so you might not need to do all of these, but this is how it would look if you did.

You will obviously need to replace the dataset name with the name of your reference catalog, and the names of the filters in the filter mapping (on each side of course).

The following should be placed in your processCcd.py config file, in your obs_package_dir/config.

from lsst.meas.algorithms import LoadIndexedReferenceObjectsTask

config.calibrate.photoRefObjLoader.retarget(LoadIndexedReferenceObjectsTask)
config.calibrate.photoRefObjLoader.ref_dataset_name = "your_dataset_name"

config.calibrate.astromRefObjLoader.retarget(LoadIndexedReferenceObjectsTask)
config.calibrate.astromRefObjLoader.ref_dataset_name = "your_dataset_name"

config.charImage.refObjLoader.retarget(LoadIndexedReferenceObjectsTask)
config.charImage.refObjLoader.ref_dataset_name = "your_dataset_name"

# Create a filterMap to map between the filters in the refCat and your simulated filters
for refObjLoader in (config.calibrate.astromRefObjLoader,
                     config.calibrate.photoRefObjLoader,
                     config.charImage.refObjLoader):
    refObjLoader.filterMap = {'PhomSimFiltername_1': 'refcat_filter_name_1',
                              'PhomSimFiltername_2': 'refcat_filter_name_2',
                              'PhomSimFiltername_3': 'refcat_filter_name_3'}

rbliu · December 7, 2017, 10:19pm

That solved the problem! It turns out I didn’t add configfile to ingestReferenceCatalog.py – that step didn’t show an error though, but processEimage.py obviously cannot recognize the ref_cat_filter names. Thank you so much @merlin !