Testing DM-4692: the new ProcessCcdTask

KSK · February 18, 2016, 4:16pm

Here’s an update on the efforts to track down the regression in performance on the CFHT data from master to the branch.

First, we noticed that there were a couple of outliers in reported astrometric residuals: ~100mas vs. ~50mas. After inspecting the images with issues and comparing between master and the branch, it was observed that the branch images were including a small number (1 or 2) of bad matches in the list passed to the fitter. It turns out that the master branch is getting around the problem because the shallow icSrc catalog is much better matched to the depth of the reference catalog than the deeper src catalog used in the branch. Having the two matched in depth reduces the chance of confusion when one of the science sources is rejected for one reason or anther: e.g. masked, just off the edge of the chip, blend.

The proposed solution to the problem is, of course, to fix the fitter to better handle outliers. In the short term and to reproduce behavior on master, Russell has implemented a user configurable S/N cut in the source catalog. Setting this to 50 as the default completely eliminates the observed outliers in reported astrometric residual.

After the proposed S/N fix, @ctslater showed that the difference between the branch and master in the reported astrometric residual is very similar with the branch being better most of the time. We still see a slight regression in the astrometric repeatability between two randomly chosen visits. This manifests in a median repeatability of 11mas for stars brighter than 21 vs. 10mas on master. Here is the relevant figure from validate_drp for all stars in the 36 CFHT chips.

I do not have the same figure for master on hand, but this thread contains a few examples of the figure for a subset of the chips in the focal plane. We are still looking into whether this difference could be explained by inherent scatter. The regression is ~1mas and the scatter in the distribution appears to be ~20mas for the bright stars.

Unfortunately, I’ll be out of touch until about 2PM Pacific, but Colin and Russell can field questions.

rowen · February 18, 2016, 7:11pm

Here are the results I measure on DM 4692 using SNR rejection.

Astrometric agreement between two visits; the bright scatter is 12% worse than on master (11.1 mas instead of 9.8) but the scatter for all sources is the same:

The astrometric fit per CCD (RMS scatter vs. number of matches) shows that there are no failed solutions, and the results look similar to master:

Photometric agreement between two visits; photometric agreement is slightly worse: 13.9 mmag instead of 13.0 mmg for bright sources and 118.5 mmag vs 116.4 mmag for all sources.

Note: the results shown above also include moving the the rejection of edge sources from the “is suitable for matching” to “is suitable fitting a WCS” test. This was done because we in a few cases with earlier code a failure mode we saw was failure to match obvious edge sources, leading to incorrect matches. However, this improvement, though motivated by logic, makes essentially no difference to the results.

rowen · February 19, 2016, 9:21pm

I tried running CFHT on master after moving the edge test. Here are the results:

I see no significant change in either plot (compare to scatter plot below, histogram above), from which I conclude that the change seems to be innocuous (we already saw this on DM-4692, though noted a slight improvement there).

rowen · February 19, 2016, 9:37pm

Here is the scatter plot for unmodified master.

boutigny · March 22, 2016, 4:14pm

I have tested the new ProcessCcdTask on 10 u band visits (I selected u band as it is providing the most demanding test).
Here is an identified issue on visit 994926 / ccd 9 (available at NCSA in /lsst8/boutigny/valid_cfht/rawDownload)

The astrometric scatter is reported to be equal 0.000 ± 0.000 arcsec 9 matches found. This is obviously an error indicating that the fit did not actually run.
Rerunning using the anet matcher gives an astrometric scatter = 0.333766 arcsec for 48 matches.
In both cases (default astrometry or anet), the photometric calibration fails with the message : “No sources remaining in match list after magnitude limit cuts”. For anet, the details of the cuts are the following : input: 11 sources - after source flag cut: 5 - after reference catalog cut: 5 - after magnitude cut: 0
With the old processCcd, the astrometry (anet) and the photometry calibration were ok (decent astrometric scatter and zero point)

Looking at the astrometric scatter for other CCDs, I see some other suspiciously low values which are also probably hiding failures.

So it seems that the default astrometric matcher and the source selection before photometry calibration require some further tuning. It also shows that we need more sophisticated validation tests to be run on new stack versions. I suggest using u band.

update : there is another instance of failed photometric calibration for visit=853091 / CCD=7 but this one has a particularly bad seeing

rowen · April 4, 2016, 5:58pm

I ran this image on my Mac and it seemed to work correctly for me. I get:

processCcd: Processing {'taiObs': '2008-05-29T10:10:11.23', 'extension': 10, 'object': 'D3', 'visit': 994926, 'filter': 'u', 'state': 'p', 'runId': '08AL04', 'date': '2008-05-29', 'ccd': 9, 'expTime': 660.165}
...
processCcd.charImage.measurePsf: Measuring PSF
processCcd.charImage.measurePsf: PSF star selector found 22 candidates
processCcd.charImage.measurePsf: PSF determination using 9/22 stars.
processCcd.charImage: iter 2; PSF sigma=2.38, dimensions=(33, 33); median background=247.82
processCcd.charImage.repair: Identified 2049 cosmic rays.
...
processCcd.calibrate.astrometry.refObjLoader: Loading reference objects using center (1023.5, 2305.5) pix = Fk5Coord(215.5813297, 52.8150844, 2000.00) sky and radius 0.13499220215 deg
processCcd.calibrate.astrometry.refObjLoader: Loaded 85 reference objects
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1340 unusable sources, leaving 56 usable sources
processCcd.calibrate.astrometry.matcher: Matched 25 sources
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1340 unusable sources, leaving 56 usable sources
processCcd.calibrate.astrometry.matcher: Matched 23 sources
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1340 unusable sources, leaving 56 usable sources
processCcd.calibrate.astrometry.matcher: Matched 21 sources
processCcd.calibrate.astrometry: Matched and fit WCS in 3 iterations; found 21 matches with scatter = 0.025 +- 0.014 arcsec
processCcd.calibrate.photoCal: Applying color terms for filterName='u', config.photoCatName=e2v because config.applyColorTerms is True
processCcd.calibrate.photoCal: Magnitude zero point: 32.261575 +/- 0.000702 from 10 stars
processCcd.calibrate: Photometric zero-point: 32.261575

This is with master as of 2016-04-01. If you are using a weekly release instead of master, would you be willing to try master?. Full log available on request (uploading text files is not allowed).

rowen · April 4, 2016, 6:22pm

I suggest we move the discussion of this failure to DM-5685 which is dedicated to the problem. I posted the full run log there.

boutigny · April 5, 2016, 9:59am

Thanks @rowen.
I confirm that processCcd is working fine on this visit / ccd for the default set of config parameters. The problem I reported is apparently due to the Chebyshev background estimation which I turned on with:
config.charImage.background.useApprox=True
config.charImage.detectAndMeasure.detection.background.binSize=128
config.charImage.detectAndMeasure.detection.background.useApprox=True
config.charImage.background.binSize = 128
config.charImage.background.undersampleStyle = 'REDUCE_INTERP_ORDER’
config.charImage.detectAndMeasure.detection.background.binSize = 128
config.charImage.detectAndMeasure.detection.background.undersampleStyle='REDUCE_INTERP_ORDER’
config.charImage.detectAndMeasure.detection.background.binSize = 128
config.charImage.detectAndMeasure.detection.background.undersampleStyle = ‘REDUCE_INTERP_ORDER’

So it may just be that these parameters are not good for the u band

RHL · April 5, 2016, 11:36am

That’s very odd. Someone should look at the background-subtracted images and see what went wrong (it should be easier not harder in u, unless there’s something funny with the electronics at low light levels)

boutigny · April 5, 2016, 12:54pm

Could @jbosch or @price check the parameters I set for Chebyshev background ? I tried to adapt this list from the one I had for the previous processCcd but I may have missed something.

jbosch · April 5, 2016, 1:43pm

@boutigny’s settings reported here match what we’re using on the HSC side. The defaults for ProcessCcd on LSST don’t seem to be using approximation, even though I know we ported that over, and I don’t even see any evidence that we’re enabling approximation in obs_subaru on the LSST side either. But that’s just from looking at the code, not dumping the config when running, so it’s possible I’m missing something, and I’m sure @laurenam or @rearmstr would have noticed a difference like this in their efforts comparing LSST-side and HSC-side processing for HSC data.

rearmstr · April 5, 2016, 1:57pm

I’ve had to enable the approximation explicitly because they are not currently the default. This is one of a couple of differences I have had to make to get HSC and LSST to agree.

price · April 5, 2016, 1:57pm

I’m rebuilding the world, and then will attempt to reproduce @boutigny’s results and post the images @RHL requested.

swinbank · April 5, 2016, 2:11pm

When @laurenam ported approximation to LSST, she found a number of bugs in afw which she reckoned made it unsafe to enable by default. See DM-2920 for details. I’m not clear if those same bugs also impact on HSC.

We were relying on a cleaned up implementation landing on DM-1991 before make things the default. It looks like that ticket has landed, so we should confirm that those issues have been fixed.

price · April 5, 2016, 3:04pm

The stars have holes in the midde, which must be causing the astrometry problems. I’ll look into what’s causing this.

price · April 5, 2016, 3:33pm

Looks to be due to CRs: the holes are masked CR,DETECTED,INTRP.

processCcd.charImage.measurePsf: PSF star selector found 21 candidates
processCcd.charImage.measurePsf: PSF determination using 6/21 stars.
processCcd.charImage: iter 2; PSF sigma=2.07, dimensions=(33, 33); median background=247.75
processCcd.charImage.repair: Identified 2121 cosmic rays.

price · April 5, 2016, 3:47pm

I believe the CR problem is due to an incorrect PSF model, which in turn is due to using a relatively bright threshold on the first detection pass. I have found in the past that because of the lower sensitivity in u-band compared to other bands, one usually wants to set charImage.detectAndMeasure.detection.includeThresholdMultiplier=1.0, so the first detection pass gets more stars (this is one reason why I’ve been in favour of removing this). It works in this case too.

price@lsst-dev:~/work/dominique $ processCcd.py input --output output --id visit=994926 ccd=9 --configfile processConfig.py --clobber-config -c charImage.detectAndMeasure.detection.includeThresholdMultiplier=1.0
: Loading config overrride file '/home/price/LSST/obs/cfht/config/processCcd.py'
: Config override file does not exist: '/home/price/LSST/obs/cfht/config/megacam/processCcd.py'
: input=/nfs/home/price/work/dominique/input
: calib=None
: output=/nfs/home/price/work/dominique/output
CameraMapper: Loading registry registry from /nfs/home/price/work/dominique/output/_parent/registry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in root: /nfs/home/price/work/dominique/output/calibRegistry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
CameraMapper: Loading Posix registry from /nfs/home/price/work/dominique/output
CameraMapper: Loading registry registry from /nfs/home/price/work/dominique/output/_parent/registry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in root: /nfs/home/price/work/dominique/output/calibRegistry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
CameraMapper: Loading Posix registry from /nfs/home/price/work/dominique/output
processCcd: Processing {'taiObs': '2008-05-29T10:10:11.23', 'extension': 10, 'object': 'D3', 'visit': 994926, 'filter': 'u', 'state': 'p', 'runId': '08AL04', 'date': '2008-05-29', 'ccd': 9, 'expTime': 660.165}
processCcd.isr: Performing ISR on sensor {'taiObs': '2008-05-29T10:10:11.23', 'extension': 10, 'object': 'D3', 'visit': 994926, 'filter': 'u', 'state': 'p', 'runId': '08AL04', 'date': '2008-05-29', 'ccd': 9, 'expTime': 660.165}
processCcd.isr: Saturation set to 70125
processCcd.charImage: Processing {'taiObs': '2008-05-29T10:10:11.23', 'extension': 10, 'object': 'D3', 'visit': 994926, 'filter': 'u', 'state': 'p', 'runId': '08AL04', 'date': '2008-05-29', 'ccd': 9, 'expTime': 660.165}
processCcd.charImage.repair: Identified 2054 cosmic rays.
processCcd.charImage.detectAndMeasure.detection: Detected 2720 positive sources to 5 sigma.
processCcd.charImage.detectAndMeasure.detection: Resubtracting the background after object detection
processCcd.charImage.detectAndMeasure.measurement: Measuring 2720 sources (2720 parents, 0 children) 
processCcd.charImage.measurePsf: Measuring PSF
/home/price/LSST/meas/algorithms/python/lsst/meas/algorithms/objectSizeStarSelector.py:372: RuntimeWarning: invalid value encountered in less
  bad = numpy.logical_or(bad, width < self.config.widthMin)
/home/price/LSST/meas/algorithms/python/lsst/meas/algorithms/objectSizeStarSelector.py:373: RuntimeWarning: invalid value encountered in greater
  bad = numpy.logical_or(bad, width > self.config.widthMax)
/home/price/LSST/meas/algorithms/python/lsst/meas/algorithms/objectSizeStarSelector.py:135: RuntimeWarning: invalid value encountered in less
  update = dist < minDist
processCcd.charImage.measurePsf: PSF star selector found 22 candidates
processCcd.charImage.measurePsf: PSF determination using 7/22 stars.
processCcd.charImage: iter 1; PSF sigma=2.31, dimensions=(33, 33); median background=247.62
processCcd.charImage.repair: Identified 2046 cosmic rays.
processCcd.charImage.detectAndMeasure.detection: Detected 2370 positive sources to 5 sigma.
processCcd.charImage.detectAndMeasure.detection: Resubtracting the background after object detection
processCcd.charImage.detectAndMeasure.measurement: Measuring 2370 sources (2370 parents, 0 children) 
processCcd.charImage.measurePsf: Measuring PSF
processCcd.charImage.measurePsf: PSF star selector found 21 candidates
processCcd.charImage.measurePsf: PSF determination using 6/21 stars.
processCcd.charImage: iter 2; PSF sigma=2.58, dimensions=(33, 33); median background=247.56
processCcd.charImage.repair: Identified 2045 cosmic rays.
processCcd.charImage.detectAndMeasure.measurement: Measuring 2370 sources (2370 parents, 0 children) 
processCcd.charImage.detectAndMeasure.measureApCorr: Measuring aperture corrections for 2 flux fields
processCcd.charImage.detectAndMeasure.measureApCorr: Aperture correction for base_PsfFlux: RMS 0.012687 from 6
processCcd.charImage.detectAndMeasure.measureApCorr: Aperture correction for base_GaussianFlux: RMS 0.007252 from 6
processCcd.charImage.detectAndMeasure.measurement: Measuring 2370 sources (2370 parents, 0 children) 
processCcd.charImage.detectAndMeasure.measurement.applyApCorr: Applying aperture corrections to 2 flux fields
processCcd.charImage.detectAndMeasure.measurement.applyApCorr: Use naive flux sigma computation
processCcd.calibrate: Processing {'taiObs': '2008-05-29T10:10:11.23', 'extension': 10, 'object': 'D3', 'visit': 994926, 'filter': 'u', 'state': 'p', 'runId': '08AL04', 'date': '2008-05-29', 'ccd': 9, 'expTime': 660.165}
processCcd.calibrate.detectAndMeasure.detection: Detected 977 positive sources to 5 sigma.
processCcd.calibrate.detectAndMeasure.detection: Resubtracting the background after object detection
processCcd.calibrate.detectAndMeasure.deblend: Deblending 977 sources
processCcd.calibrate.detectAndMeasure.deblend: Deblended: of 977 sources, 86 were deblended, creating 377 children, total 1354 sources
processCcd.calibrate.detectAndMeasure.measurement: Measuring 1354 sources (977 parents, 377 children) 
/ssd/price/lsstsw/miniconda/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/ssd/price/lsstsw/miniconda/lib/python2.7/site-packages/numpy/core/_methods.py:70: RuntimeWarning: invalid value encountered in true_divide
  ret = ret.dtype.type(ret / rcount)
processCcd.calibrate.detectAndMeasure.measurement.applyApCorr: Applying aperture corrections to 2 flux fields
processCcd.calibrate.detectAndMeasure.measurement.applyApCorr: Use naive flux sigma computation
processCcd.calibrate: Copying flags from icSourceCat to sourceCat for 773 sources
processCcd.calibrate.astrometry.refObjLoader: Loading reference objects using center (1023.5, 2305.5) pix = Fk5Coord(215.5813297, 52.8150844, 2000.00) sky and radius 0.13499220215 deg
processCcd.calibrate.astrometry.refObjLoader: Loaded 85 reference objects
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1314 unusable sources, leaving 40 usable sources
processCcd.calibrate.astrometry.matcher: Matched 23 sources
processCcd.calibrate.astrometry.wcsFitter: Updating centroids in refCat
processCcd.calibrate.astrometry.wcsFitter: Updating coords in sourceCat
processCcd.calibrate.astrometry.wcsFitter: Updating distance in match list
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1314 unusable sources, leaving 40 usable sources
processCcd.calibrate.astrometry.matcher: Matched 22 sources
processCcd.calibrate.astrometry.wcsFitter: Updating centroids in refCat
processCcd.calibrate.astrometry.wcsFitter: Updating coords in sourceCat
processCcd.calibrate.astrometry.wcsFitter: Updating distance in match list
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1314 unusable sources, leaving 40 usable sources
processCcd.calibrate.astrometry.matcher: Matched 21 sources
processCcd.calibrate.astrometry.wcsFitter: Updating centroids in refCat
processCcd.calibrate.astrometry.wcsFitter: Updating coords in sourceCat
processCcd.calibrate.astrometry.wcsFitter: Updating distance in match list
processCcd.calibrate.astrometry: Matched and fit WCS in 3 iterations; found 21 matches with scatter = 0.037 +- 0.021 arcsec
processCcd.calibrate.photoCal: Applying color terms for filterName='u', config.photoCatName=e2v because config.applyColorTerms is True
processCcd.calibrate.photoCal: Magnitude zero point: 32.261447 +/- 0.000699 from 10 stars
processCcd.calibrate: Photometric zero-point: 32.261447

RHL · April 18, 2016, 3:04am

Labelling bright stars as CRs is always a danger.

If you have a list of stars you can tune the CR parameters to not find their cores, and that was basically what I did in SDSS. I think that would work here too.

timj · April 18, 2016, 5:32pm

Do you mean you feed the catalog to the CR algorithm and it knows to take special care when it is near a bright star? Or do you mean that you use the prior knowledge to adjust the overall parameters to use for a particular field?

RHL · April 19, 2016, 12:46am

You tune the CR code to not think that the cores of the PSF stars are CRs