Source injection variance

Support
#22

The gain and the units are independent, and operate differently on image and variance plane.

The gain tells you how it was converted from counts to electrons (note that the pvi has electron units so the gain has been applied; the effective gain of the image is 1.0!). And then there’s a straight unit conversion to nJy given by the photocalib.

But the way this is recorded is indeed confusing, especially for injection, now that you put it this way. The gain tells you how it was processed, but the photocalib object tells you how to convert to nJy. We don’t have a way of recording how the image was converted to nJy, which is part of the reason why we don’t put the pvi in nJy-like-units.

#23

Thanks @lskelvin and @erykoff ! I am a bit confused about the empirical variance estimation though. We don’t measure the variance, it is computed from some calibration on the number of photons/electrons vs the output ADU of the camera, right? So there should be a number somewhere in the analysis pipeline (or a series of numbers that need to be multiplied) that directly scales flux to variance. Extra variance is added by some non-signal dependent sources, but somewhere in the pipeline the Poisson variance is being inferred from the measured signal. I don’t see what “empirical” aspect we can lean on to get the variance scaling. I know that one can look at dark patches in an image to get an estimation of the noise floor, but that doesn’t really give Poisson noise scaling.

#24

@ConnorStone Part of the answer to your question sits behind a bit of code that @Alex-Broughton pointed us to above. During the production of preliminary_visit_image, IsrTaskLSST computes an initial value for the variance plane of the image. The way it does this, according to the addVariancePlane code, starts by checking whether the exposure units are “electron” (which it looks up in exposure.metadata[“LSST ISR UNITS”]). Assuming this is true (it is for DP1 visit_images), it sets a variable named gain to 1.0. It also consults the exposure metadata to get the readNoise for each amplifier, stored in “LSST ISR READNOISE {amp.getName()}”.

Having set these numbers, it passes them and the exposure to updateVariance in isrFunctions.py. This takes whatever the preexisting variance plane is (which I’m hoping is just all zeros) and adds to it the image flux in each pixel, multiplied by 1/gain. This is just the Poisson term. Then it adds (readNoise/gain)**2 to each pixel. Since we have gain == 1.0, the variance at this stage is just equal to the image plus readNoise**2.

To be clear: if we make a plot of variance vs. flux in each pixel (in a single amplifier region of a maskedImage), at this stage all the points should lie perfectly along a straight line, with a slope of 1.0 and a y-intercept of readNoise**2.

Now, as Alex and @erykoff pointed to above, the flat fielding then modifies both the image and the variance plane in place, hopefully in a consistent manner. This is a scale factor that will generally change the “effective” gain. My understanding is that every pixel is scaled differently by the flat field correction, which causes the variance vs. flux scatterplot to no longer be a perfectly straight line.

If, as @erykoff asserts, the only other main thing that needs to be taken into account is the flat fielding - and if our image processing pipeline allows flat fielding to be an invertible operation - then if we invert the flat fielding correction, I would expect the effective gain to go back to 1.0. If we can only approximately invert it because of other stuff that happens in visit_image processing, I would expect the effective gain to be approximately 1.0. In practice the effective gain far away from this, closer to 0.4, even after inverting the flat field correction. This is a puzzle that I’d still like to figure out.

This may mainly come down to a units issue as @lskelvin noted above. If the change in units from electrons to nJy should be reflected in the getExposureGains, then if these are not being updated correctly, this may not be solvable in our current code.

When @lskelvin is talking about “empirical” gain estimation he’s referring to estimating the effective gain of the final visit_image by making a simple straight-line fit of variance vs. flux in each pixel. I’ll note that some of the variance vs. flux plots display some pronounced nonlinearities in at least some pixels, even after inverting as many steps of visit_image processing as I know how to do, so there’s some level of risk we have to accept with this method.

1 Like
#25

Thanks @jjbuchanan, @erykoff, @lskelvin, and @Alex-Broughton for the investigation! @ConnorStone, I am checking in to see if the latest response by @jjbuchanan answers and resolves your questions.

#26

Hi @galaxyumi331, I think some really good progress has been made, but it doesn’t seem like the source of the incorrect variance has been truly nailed down. See, e.g., @jjbuchanan’s comment:

…I would expect the effective gain to be approximately 1.0. In practice the effective gain far away from this, closer to 0.4, even after inverting the flat field correction. This is a puzzle that I’d still like to figure out.

I’d like to keep the discussion going until the issue is solved and Source Injection is functioning properly. If there is a more appropriate place to host the discussion I’d be happy to move things there.

#27

Thanks @ConnorStone – this is definitely the right place for the conversation! Rubin Community Science team members (like Yumi and myself) monitor the Forum, and will continue to check in periodically until a solution is marked. But please do continue the discussion.

1 Like
#28

At this stage it looks like our immediate-term strategy will be to use the “empirical” variance. I will get to work on coding this up - or rather, simplifying our existing code, because it unifies how we do things in coadds and visit_images.

While I am hoping that with further effort we will be able to definitively reconstruct the unit conversions (and anything else of relevance) as they happen in the pipelines, Lee makes a good point that the empirical variance may be the safest approach in general, as it would naturally accommodate any future changes to the pipelines.