The new ProcessCcdTask is ready to be tested on real data. This topic is intended as a place for help and to record feedback and questions. I would like to emphasis the algorithms and proper values for configuration parameters here (minor code issues are probably best discussed on the ticket or as github comments). Please see DM-4692 itself for a description of how the new ProcessCcdTask works.
Some settings that may be of interest when testing on real data:
charImage.psfIterations: number of iterations to converge on a PSF solution
charImage.detectAndMeasure.detection.includeThresholdMultiplier and thresholdValue
Te product gives the # sigma for detecting sources used to fit the PSF
and which appear in the icSrc catalog
calibrate.detectAndMeasure.detection.includeThresholdMultiplier and thresholdValue
(the product gives the # sigma for detecting sources used to fit WCS and zero-point
and which appear in the Src catalog
The long-term plan is to use only one bin/processCcd.py, but for now SDSS and the version of DECam that uses output from the community pipeline will both need trivial variants that set the dataset type of the --id argument (e.g. for SDSS the value is fpC). Simply retargeting the ISR task is not sufficient. obs_sdss has the needed code.
Two visits of raw DECam data are processed through ISR and processCcd.py using stack b1884 and the ticket branch (built on late afternoon Jan 29). They are Strire 82 data in r band, and the sdss-dr8 astrometry_net_data is used. After ISR, the default configuration parameters are used. These are the check_astrometry.png plots generated by validateDecam.py in validate_drp.
FYI: it turns out I had the deblender disabled. I have pushed a change that enables it for CalibrateTask, but not CharacterizeImageTask.
Unfortunately, it seems to be overly aggressive and I have not yet figured out why. As far as I can see the settings are identical between master and DM 4692 yet the deblender on DM 4692 finds many more children and some of them cause problems such as “Input shape is singular” and runtime errors from calcmom.
On known difference is that master runs astrometry and photoCal before the final deep detection and deblending pass. But both old and new fit a PSF model before deblending.
The difference may be appearing before we get to the deblender: an aggressive deblender could actually just be more peaks being found in detection, which could come from convolving the image by the wrong PSF (probably one too small) during detection.
At risk of speaking for @hsinfang I agree that DM 4692 results are better. Unfortunately that was with the deblender disabled, and as noted above, the deblender is not yet behaving well for DM 4692. Using the deblender results in a faint plot that is a mess and a bright plot that is fairly similar to master.
Right, they are not the best comparison as the deblender was on in the first plot and off in the second. I will rebuild the ticket branch with the newer default and rerun.
I fixed two significant errors in the last few days (with thanks to @price for identifying them):
Detection was not using smoothing. This is what caused the overly aggressive deblending.
Aperture correction was being mis-measured due to using the wrong test for suitable candidate sources. This was partly caused by having not yet implemented copying of a few important fields from icSrc to src catalog.
At this point I think the results from ProcessCcdTask should be fine. Please give it a try.
Aperture correction still needed more work. As of this morning I believe it is working. There are significant changes to pipe_tasks and a minor change to obs_decam (including allowing more cosmic rays, as one image in validate_drprunDecamTest.sh was exceeding the limit).
If you get a complaint: AttributeError: 'DetectAndMeasureConfig' object has no attribute 'starSelector' this probably means that you have an outdated config file (indeed DetectAndMeasure no longer has its own star selector; this is part of the fix to measuring aperture correction).
Running runDecamTest.sh the median astrometric error for bright sources has dropped from 61.7 mas (running master) to 49.0 mas.
One worrisome thing I found: the output of lsst_dm_stack_demo now includes some sources whose RA/Dec are nans. (It also includes more sources than before.)
Here is another, worse: validate_drp’s examples/runCfthTest.sh is giving worse astrometric errors, and the graphs show high errors even for bright sources, which is new and unsettling. Something is wrong.
@hsinfang so the branch gives slightly worse astrometric scatter for all sources, and slightly better for bright sources. If you still have the run log available for the branch, one thing to look at is whether the branch finds a reasonable number of matches for all CCDs (this shows up near the end). I find that most of the CCDs look fine, but ccd 12 has only 50 matches, instead of over 1200.
@rowen For the above plots I only processed the northern CCDs, so ccd 12 is not inlcude
d. The matches between two visits (output from validateDecam.py) are between ~1200-1800 for all northern CCDs.
Yesterday @ctslater tracked down the problem that the output of the stack demo included sources with NaNs in positions. This was a bug in how I was copying icSrc files to src (thanks to @price and @jbosch for the solution). That is now fixed.
In addition I pushed minor changes to obs_cfht master’s config overrides for processCcd: this eliminates all remaining need to use overrides when running validate_drp on master or DM-4692. However, until DM-5005 is implemented, avoiding those overrides requires manually emptying the override files in validate_drp’s config directory or processing the data manually using commands such as the following:
this is safer than using the examples in any case, due to the fact that the examples delete all data in DECam and Cfht before running (so those are not a safe location to save data from previous runs).
These changes had no effect on the output of validate_drp, as far as I can see. So the earlier graphs still apply and the mystery remains: why is the astrometric scatter reported by validate_drp somewhat worse using DM-4692 than using master? This result is obtained by comparing the catalogs of two different fields.
Making the astrometry task ignore deblended sources improved the astrometric fit significantly for CFHT on DM 4692; the scatter plot looks much better, though the results are still not as good as master, and there is a small noise floor for bright sources on DM 4692.
