The following are from half of the DECam (northern CCDs only) with the same two visits using the branch and master (version of this morning)
branch:
master
@hsinfang so the branch gives slightly worse astrometric scatter for all sources, and slightly better for bright sources. If you still have the run log available for the branch, one thing to look at is whether the branch finds a reasonable number of matches for all CCDs (this shows up near the end). I find that most of the CCDs look fine, but ccd 12 has only 50 matches, instead of over 1200.
@rowen For the above plots I only processed the northern CCDs, so ccd 12 is not inlcude
d. The matches between two visits (output from validateDecam.py) are between ~1200-1800 for all northern CCDs.
Reran the same DECam data with the rebased branch and newer master. This time both northern and southern CCDs are included.
branch
master
Yesterday @ctslater tracked down the problem that the output of the stack demo included sources with NaNs in positions. This was a bug in how I was copying icSrc files to src (thanks to @price and @jbosch for the solution). That is now fixed.
In addition I pushed minor changes to obs_cfht master’s config overrides for processCcd: this eliminates all remaining need to use overrides when running validate_drp on master or DM-4692. However, until DM-5005 is implemented, avoiding those overrides requires manually emptying the override files in validate_drp’s config directory or processing the data manually using commands such as the following:
processCcdDecam.py DECam/input --output=decam_saved/dm4692 --id -j 8 --logdest=dm4692_log.txt
processCcd.py CFHT/input --output=cfht_saved/master --id -j 8 --logdest=master_log.txt
this is safer than using the examples in any case, due to the fact that the examples delete all data in DECam and Cfht before running (so those are not a safe location to save data from previous runs).
These changes had no effect on the output of validate_drp, as far as I can see. So the earlier graphs still apply and the mystery remains: why is the astrometric scatter reported by validate_drp somewhat worse using DM-4692 than using master? This result is obtained by comparing the catalogs of two different fields.
The current results from validate_drp are as follows:
CFHT astrometric scatter (comparing two visits):
- master: 9.8 mas bright sources, 59.7 mas all sources
- DM-4692: 12.4 mas bright sources (20% worse), 64.3 mas all sources (7% worse)
CFHT photometric scatter:
- master: 13.2 mmag bright sources, 116.4 mmag all sources
- DM-4692: 14.3 mmag bright sources (8% worse), 119.2 mmag all sources (3% worse)
DECam astrometric scatter:
- master: 35.7 mas bright sources, 72.4 mas all sources
- DM-4692: 28.5 mas bright sources (20% better), 66.8 mas all sources (8% better)
DECam photometric scatter:
- master: 63.3 mmag bright sources, 81.2 mmag all sources
- DM-4692: 69.3 mmag bright sources (9% worse), 86.0 mmag all sources (6% worse)
Making the astrometry task ignore deblended sources improved the astrometric fit significantly for CFHT on DM 4692; the scatter plot looks much better, though the results are still not as good as master, and there is a small noise floor for bright sources on DM 4692.
Here’s an update on the efforts to track down the regression in performance on the CFHT data from master to the branch.
First, we noticed that there were a couple of outliers in reported astrometric residuals: ~100mas vs. ~50mas. After inspecting the images with issues and comparing between master and the branch, it was observed that the branch images were including a small number (1 or 2) of bad matches in the list passed to the fitter. It turns out that the master branch is getting around the problem because the shallow icSrc catalog is much better matched to the depth of the reference catalog than the deeper src catalog used in the branch. Having the two matched in depth reduces the chance of confusion when one of the science sources is rejected for one reason or anther: e.g. masked, just off the edge of the chip, blend.
The proposed solution to the problem is, of course, to fix the fitter to better handle outliers. In the short term and to reproduce behavior on master, Russell has implemented a user configurable S/N cut in the source catalog. Setting this to 50 as the default completely eliminates the observed outliers in reported astrometric residual.
After the proposed S/N fix, @ctslater showed that the difference between the branch and master in the reported astrometric residual is very similar with the branch being better most of the time. We still see a slight regression in the astrometric repeatability between two randomly chosen visits. This manifests in a median repeatability of 11mas for stars brighter than 21 vs. 10mas on master. Here is the relevant figure from validate_drp for all stars in the 36 CFHT chips.
I do not have the same figure for master on hand, but this thread contains a few examples of the figure for a subset of the chips in the focal plane. We are still looking into whether this difference could be explained by inherent scatter. The regression is ~1mas and the scatter in the distribution appears to be ~20mas for the bright stars.
Unfortunately, I’ll be out of touch until about 2PM Pacific, but Colin and Russell can field questions.
Here are the results I measure on DM 4692 using SNR rejection.
Astrometric agreement between two visits; the bright scatter is 12% worse than on master (11.1 mas instead of 9.8) but the scatter for all sources is the same:
The astrometric fit per CCD (RMS scatter vs. number of matches) shows that there are no failed solutions, and the results look similar to master:
Photometric agreement between two visits; photometric agreement is slightly worse: 13.9 mmag instead of 13.0 mmg for bright sources and 118.5 mmag vs 116.4 mmag for all sources.
Note: the results shown above also include moving the the rejection of edge sources from the “is suitable for matching” to “is suitable fitting a WCS” test. This was done because we in a few cases with earlier code a failure mode we saw was failure to match obvious edge sources, leading to incorrect matches. However, this improvement, though motivated by logic, makes essentially no difference to the results.
I tried running CFHT on master after moving the edge test. Here are the results:
I see no significant change in either plot (compare to scatter plot below, histogram above), from which I conclude that the change seems to be innocuous (we already saw this on DM-4692, though noted a slight improvement there).
I have tested the new ProcessCcdTask on 10 u band visits (I selected u band as it is providing the most demanding test).
Here is an identified issue on visit 994926 / ccd 9 (available at NCSA in /lsst8/boutigny/valid_cfht/rawDownload)
-
The astrometric scatter is reported to be equal 0.000 ± 0.000 arcsec 9 matches found. This is obviously an error indicating that the fit did not actually run.
-
Rerunning using the anet matcher gives an astrometric scatter = 0.333766 arcsec for 48 matches.
-
In both cases (default astrometry or anet), the photometric calibration fails with the message : “No sources remaining in match list after magnitude limit cuts”. For anet, the details of the cuts are the following : input: 11 sources - after source flag cut: 5 - after reference catalog cut: 5 - after magnitude cut: 0
-
With the old processCcd, the astrometry (anet) and the photometry calibration were ok (decent astrometric scatter and zero point)
Looking at the astrometric scatter for other CCDs, I see some other suspiciously low values which are also probably hiding failures.
So it seems that the default astrometric matcher and the source selection before photometry calibration require some further tuning. It also shows that we need more sophisticated validation tests to be run on new stack versions. I suggest using u band.
update : there is another instance of failed photometric calibration for visit=853091 / CCD=7 but this one has a particularly bad seeing
I ran this image on my Mac and it seemed to work correctly for me. I get:
processCcd: Processing {'taiObs': '2008-05-29T10:10:11.23', 'extension': 10, 'object': 'D3', 'visit': 994926, 'filter': 'u', 'state': 'p', 'runId': '08AL04', 'date': '2008-05-29', 'ccd': 9, 'expTime': 660.165}
...
processCcd.charImage.measurePsf: Measuring PSF
processCcd.charImage.measurePsf: PSF star selector found 22 candidates
processCcd.charImage.measurePsf: PSF determination using 9/22 stars.
processCcd.charImage: iter 2; PSF sigma=2.38, dimensions=(33, 33); median background=247.82
processCcd.charImage.repair: Identified 2049 cosmic rays.
...
processCcd.calibrate.astrometry.refObjLoader: Loading reference objects using center (1023.5, 2305.5) pix = Fk5Coord(215.5813297, 52.8150844, 2000.00) sky and radius 0.13499220215 deg
processCcd.calibrate.astrometry.refObjLoader: Loaded 85 reference objects
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1340 unusable sources, leaving 56 usable sources
processCcd.calibrate.astrometry.matcher: Matched 25 sources
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1340 unusable sources, leaving 56 usable sources
processCcd.calibrate.astrometry.matcher: Matched 23 sources
processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 85 stars
processCcd.calibrate.astrometry.matcher: Purged 1340 unusable sources, leaving 56 usable sources
processCcd.calibrate.astrometry.matcher: Matched 21 sources
processCcd.calibrate.astrometry: Matched and fit WCS in 3 iterations; found 21 matches with scatter = 0.025 +- 0.014 arcsec
processCcd.calibrate.photoCal: Applying color terms for filterName='u', config.photoCatName=e2v because config.applyColorTerms is True
processCcd.calibrate.photoCal: Magnitude zero point: 32.261575 +/- 0.000702 from 10 stars
processCcd.calibrate: Photometric zero-point: 32.261575
This is with master as of 2016-04-01. If you are using a weekly release instead of master, would you be willing to try master?. Full log available on request (uploading text files is not allowed).
I suggest we move the discussion of this failure to DM-5685 which is dedicated to the problem. I posted the full run log there.
Thanks @rowen.
I confirm that processCcd is working fine on this visit / ccd for the default set of config parameters. The problem I reported is apparently due to the Chebyshev background estimation which I turned on with:
config.charImage.background.useApprox=True
config.charImage.detectAndMeasure.detection.background.binSize=128
config.charImage.detectAndMeasure.detection.background.useApprox=True
config.charImage.background.binSize = 128
config.charImage.background.undersampleStyle = 'REDUCE_INTERP_ORDER’
config.charImage.detectAndMeasure.detection.background.binSize = 128
config.charImage.detectAndMeasure.detection.background.undersampleStyle='REDUCE_INTERP_ORDER’
config.charImage.detectAndMeasure.detection.background.binSize = 128
config.charImage.detectAndMeasure.detection.background.undersampleStyle = ‘REDUCE_INTERP_ORDER’
So it may just be that these parameters are not good for the u band
That’s very odd. Someone should look at the background-subtracted images and see what went wrong (it should be easier not harder in u, unless there’s something funny with the electronics at low light levels)
Could @jbosch or @price check the parameters I set for Chebyshev background ? I tried to adapt this list from the one I had for the previous processCcd but I may have missed something.
@boutigny’s settings reported here match what we’re using on the HSC side. The defaults for ProcessCcd on LSST don’t seem to be using approximation, even though I know we ported that over, and I don’t even see any evidence that we’re enabling approximation in obs_subaru on the LSST side either. But that’s just from looking at the code, not dumping the config when running, so it’s possible I’m missing something, and I’m sure @laurenam or @rearmstr would have noticed a difference like this in their efforts comparing LSST-side and HSC-side processing for HSC data.
I’ve had to enable the approximation explicitly because they are not currently the default. This is one of a couple of differences I have had to make to get HSC and LSST to agree.