detectCoaddSource detection threshold on noisy background data

Support
#1

Hello,

We are trying to run detectCoaddSource on emulated HSC Coadd with VISTA data. The background is not well subtracted. This leads to a huge correction of the inputed threshold and the increase of false detection in the end. We would like to know if there is a way not to modify the configured detection threshold after the detection of the sky sources like it is done in this example ?

detectCoaddSources.py . --rerun RERUN --id tract=9813 patch=4,4 filter=VIRCAM-J --clobber-config --clobber-version --config detection.thresholdValue=5.0 doScaleVariance=True

detectCoaddSources.scaleVariance INFO: Renormalizing variance by 0.856342
detectCoaddSources.detection INFO: Applying temporary wide background subtraction
detectCoaddSources.detection INFO: Detected 44011 positive peaks in 4645 footprints to 2.5 sigma
detectCoaddSources.detection.skyObjects INFO: Added 528 of 1000 requested sky sources (53%)
detectCoaddSources.detection.skyMeasurement INFO: Performing forced measurement on 528 sources
detectCoaddSources.detection INFO: Modifying configured detection threshold by factor 0.327976 to 1.639878
detectCoaddSources.detection INFO: Tweaking background by -0.307181 to match sky photometry
detectCoaddSources.detection INFO: Detected 32690 positive peaks in 3064 footprints to 1.63988 sigma
detectCoaddSources.detection INFO: Detected 63323 positive peaks in 2028 footprints to 1.63988 sigma
detectCoaddSources.detection.skyObjects INFO: Added 170 of 1000 requested sky sources (17%)
detectCoaddSources.detection.skyMeasurement INFO: Performing forced measurement on 170 sources
detectCoaddSources.detection INFO: Tweaking background by -0.246550 to match sky photometry

Also, we noticed in our tests that the default setting for config.detection.thresholdType=‘pixel_stdev’ but the results improve when set to ‘stdev’ with the same detection threshold. What is the difference between the two exactly ?

Thanks a lot in advance for your help.

Guillaume D

#2

Putting something like this in a configuration file will switch to a simpler detection algorithm that does not try to update the threshold:

from lsst.meas.algorithms import SourceDetectionTask
config.detection.retarget(SourceDetectionTask)

I believe pixel_stdev uses a local (per-pixel) estimate of the variance, while stdev uses a global one. I imagine the former works better only when the variance really does change appreciably over the image and the per-pixel estimate (which is done at the earliest stages of processing and then propagated through to the coadd) is robust.

#3

I think the two of these together are telling you that the variance plane in your data is very inaccurate.

#4

I’m not sure the variance map is the culprit. I think the issue is that the pixel-based variance and the large-scale variance (from the sky apertures) are inconsistent:

See for exemple the variance rescaling (which I suspect is done on pixels):
detectCoaddSources.scaleVariance INFO: Renormalizing variance by 0.856342
and doesn’t indicate any fundamental issue with the per-pixel variance.

Now, after computing the sky rms from random aperture, the threshold is significantly modified which I suspect comes form the background structures in the image (not 100% sure though).

If that’s the case, iIt will lead to biased flux measurements I believe, so I’m not saying we’re happy with these data (far from it), but at least we would like to fix the detection threshold so that we have meaningful objects to do tests on.

@price do you want to have a look at the data?

#5

That could be the case, if the background subtraction is bad, as you say it is.

Could you post some images here, please?

#6

@desprezg can you provide the corresponding images?

#7

Here is an exemple of a patch we are working on : https://www.dropbox.com/sh/4tb2bx9jztu9rzn/AACLKXwq-2-rNt--BL68P1BHa?dl=0

I ran the detection like this :

detectCoaddSources.py . --rerun s18a_udeep_deep_depth.ext_v1.0 --id tract=8524 patch=4,6 filter=VIRCAM-H --clobber-config --clobber-version --config detection.thresholdValue=5.0 doScaleVariance=True detection.thresholdType='pixel_stdev'
#8

Having looked at the image and code, I suspect that the threshold tweaks are correct (because the background gets removed in the threshold tweak calculation), but when the threshold is lowered the background structures blend everything together — you can’t get detection of faint astrophysical sources if the sky variations are larger. I think you’re going to need to increase the threshold. Using thresholdType=stdev and/or the regular SourceDetectionTask are two ways to do that, as is increasing the thresholdValue.

#9

@jbosch says this is a “global” value but what does that mean exactly? That the image pixel value is compared to the mean (or something) of the variance map? As opposed to the corresponding variance pixel? I’m a bit confused. Where can we find the documentation on the detection parameters?

#10

thresholdType=stdev means that the threshold is set from the standard deviation of the image pixels: the variance plane doesn’t come into it.
thresholdType=pixel_stdev means that the threshold is set from dividing the image plane by the square root of the variance plane.

There are some brief explanations in the configuration doc.