During the processCcd stage I am struggling to get any CCDs to pass without throwing the ‘No PSF candidates survive error’.
Interestingly if I run on six stacked exposures I can get the stacks to find enough psf candidates. I have tried changing config for both the star selection and psf determination:
Star selection usually finds a handful of candidates (~10) but none of these survive. I have experimented with setting all requirements to as lenient a value as appears possible to me.
Does anybody have any advice for how to change configuration options to allow more PSF candidates to survive the PSF determination stage in shallower exposures?
Here is a screenshot comparison of the exposures which fail (left) and the six exposure stacks that pass (right) which have also had some further artefact removal:
It looks like the psfDeterminer is throwing that error, but I expect because the selector is selecting too few to begin with. I’m no expert on PSF characterisation, but I’d worry that ~ten PSF candidates per frame wouldn’t be enough to characterise the PSF well, even if they did all survive. So, I think you want to work on the selector first to boost the numbers getting through to the PCA.
Do you know how many sources are actually being detected in each frame, i.e., prior to the objectSize selection?
I don’t believe anyone has used the pca psfDeterminer in earnest for quite a long time. We use psfex (via the meas_extensions_psfex package) for HSC, and I believe that every other camera has followed our lead on this.
There are config parameters in the starSelector that sets the minimum flux (fluxMin, signalToNoiseMin) that you should play with.
But I think the root cause of the problem is that you simply don’t have a lot of suitable PSF stars to work with in the left-hand image. How would you generate a PSF model from this image if you weren’t using the LSST stack?
I think in the orginal pipeline they estimate the PSF from the stack. We are considering doing this also but I have been informed that due to dithering between exposures and the fact we undersample the psf this could be suboptimal. I wanted to try running on the exposures to try to quantify differences. We may well end up importing the six exposure stacks as if they were exposures but I’m nervous of what effects this will have.
If you measure the PSF from a stack, it won’t have the same fidelity as if you measured the PSF on the individual images. But if you can’t measure the PSF from an individual image because of low signal-to-noise, you’d be forced to guess at a PSF, which means you’re not going to have a good stack PSF either. I think you’re caught between a rock and a hard place, and no matter what you choose, you’re just not going to have an A1 quality PSF. So you may as well make it easy for yourself and work with the stacked images.
If you did want to work with coadded images – and get the stack to coadd them, as opposed to ingest your own coadds – then you could take a look at obs_goto’s singleVisitDriver.py, which we wrote to do exactly that. It astrometrically registers, warps, and combines back-to-back exposures before running charImage on the result.
The way it’s written, you’d need a reliable way to identify the exposures that you want to coadd. In our case, we were able to use our visit number, since a set of three back-to-back exposures were tagged with the same visit number.
The VISTA pipeline already has the 6 exposure stacks everywhere so I can use them. Conceivably we might want to reproduce these so the singleVisitDriver might be useful thanks. I am going to try to push the Star Selector to find more sources. I have pasted the full numbers below for a typical exposure. I can increase the number selected but not get it to pass the determination stage.
Thanks for your ongoing help.
Raphael.
processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processCcd.charImage.detection INFO: Detected 686 positive peaks in 382 footprints and 220 negative peaks in 178 footprints to 50 sigma
processCcd.charImage.detection INFO: Resubtracting the background after object detection
processCcd.charImage.measurement INFO: Measuring 382 sources (382 parents, 0 children)
processCcd.charImage.measurePsf INFO: Measuring PSF
processCcd.charImage.measurePsf INFO: PSF star selector found 299 candidates
processCcd.charImage.measurePsf.reserve INFO: Reserved 0/299 sources
processCcd.charImage.measurePsf INFO: Sending 299 candidates to PSF determiner
processCcd.charImage.measurePsf.psfDeterminer WARN: Please provide at least one Image for me to update: reducing number of eigen components
processCcd.charImage.measurePsf.psfDeterminer WARN: Please provide at least one Image for me to update: reducing number of eigen components
processCcd.charImage.measurePsf.psfDeterminer WARN: Please provide at least one Image for me to update: reducing number of eigen components
processCcd FATAL: Failed on dataId={'dateObs': '2017-10-27', 'visit': 614753, 'obsNum': '00133', 'filter': 'VISTA-Y', 'hdu': 2, 'ccd': 1, 'dataType': 'BINTABLE', 'expTime': 30.0, 'taiObs': '2017-10-28T02:03:30.899', 'ccdnum': 1}: IndexError: No viable PSF candidates survive
220 negative peaks sounds rather large, and comparable to the number of positive peaks, so probably you’re getting a lot of noise. I suspect your variance map is inaccurate.
In regular processCcd use, a variance map is constructed during ISR. If you’re overriding ISR, then you need to build a variance map yourself. If you’re using the regular ISR, then you’ve probably got the gain and readnoise set incorrectly (and/or there’s a lot of covariance, which we don’t track).
Ok thank you. I think I have switched off ISR but will look into what is happening there. I also have variance maps from the six exposure stacks but am not sure how to ingest them. Is there an example of this in any of the other obs packages?
I don’t think you can ingest the variances separately without doing some deep magic (though maybe the Gen3 butler makes it easier?).
But ultimately, I think you want something similar to the DecamNullIsrTask, except that instead of just reading and returning the instcal dataset (or whatever you want to call your pre-processed data), you also set the exposure.variance to your known variance map.
Hi @raphaelshirley, I am just following up on unsolved topics in Support. It looks like, in this case, some of the suggestions unearthed some underlying issues that have been pursued in other topics: " The obs_decam nullIsr task" and " Negative peaks in calexp files". I was wondering if (for posterity) there are any final comments or takeaways for anyone else who might have similar issues in finding and using PSF stars in low SNR images?
Yes this has been solved by fully implementing the DecamNulISR task. There is further discussion regarding how to implement it with the gen 2 Butler here. The three aspects of the pstISRCCD (map, mask, and variance map) must be tied together and propagated past the ISR stage.
