I’m trying to get my head around what’s going on in the pipe_tasks/photoCal.py as the uncertainties we’re getting on our mags are too small (compared to their standard deviation from reference sources), so I’m putting out a request for a little help.
As I understand it from the top line description in getZeroPoint, it should return the zeropoint, the width of the (src - ref) distribution, and the number of good points (after sigma clipping). Indeed, that does seem to what happens if ngood == 0 at line 689. Otherwise, the sigma it returns is the inverse sqrt of sum(1/error**2), where error is, I believe, the error due to photon counting statistics. I’m struggling to work out how the latter gives a good estimate of the width of the (src-ref) distribution.
I can see how the latter would hold if photon statistics were the only source of errors, but that’s rarely (never?) the case in my experience!
We get far more appropriate errors if we instead adopt sigma=sig on line 707, which would return an estimate of the width of the (src-ref) distribution (as on line 689).
I cannot answer your immediate question above, but I’ll note that there are several other questions about the error propagation code in PhotoCalTask. See e.g. DM-2308 (reference flux errors are unused) and DM-17692 (uncertainty after colorterms is questionable). A full audit of that algorithm may be a good idea.
No problems. Thanks for the quick response, John. We’re in the process of retargeting our own PhotoCalTask in which sigma=sigma at line 707 is replaced with sigma=sig, since the latter seems (at least to us) to give a more reasonable measure of the distribution width. We’ll run some tests and report back, if people think it may be insightful.
I think much of the issue here is that we’ve never really been consistent about what the uncertainty reported by PhotoCal is supposed to be for. Most (I think all) downstream processing steps simply ignore it, but the classes we use to represent the photometric calibration have historically treated it as a source of uncertainty that should be propagated into the calibrated fluxes. That’s clearly not what you’d do if the uncertainty was an empirical measure of the distribution, and I think we’ve recently come to the conclusion that it’s not something you can naively do if it’s just derived propagation of photon-noise errors from the photometry that went into the photometric fit, because of course that makes it so the original photomeric errors and the photometric calibration errors are not independent.
In any case, if what you’re looking is a measure of the overall photometric error budget, what you’re doing sounds reasonable. I suspect the right longer-term fix for the pipeline code is to stop trying to propagate uncertainty in the calibration into individual calibrated fluxes at all, and explicitly (and separately) report both the empirical width of the distribution and the expected uncertainty in the calibration from propagating measurement errors (and possibly some other estimates of the uncertainty from some kind of cross-validation).
Thanks, Jim, for this detailed explanation. Yes - I see what you mean about how this could lead to (for want of a better phrase) “double-count” the photon noise in the uncertainties, so I now understand why it’s currently written as it is. In our case, I think we’re a long way from the regime where this “double-counting” would be a problem, so it does seem more reasonable (as you say) to use sig as a measure of the overall photometric error budget.
Given what you and John both say, it sounds like this is going to be something that will go through some quite major changes in the future. We’ll keep an eye open for updates, but since our simple tests are indicating that taking the sig gives a much better (although not perfect) representation of the overall photometric uncertainties we’ll stick with our simple fix for now; I don’t fancy trying to implement the “longer-term” fix you suggest - I’ll leave that to you experts!
Again, thanks for your help in clarifying what was going on in PhotoCal.