Hi all,
I know that normally detectCoaddSources.py is run as part of coaddDriver.py, but I need to run after emulating an external image and before running the multi band process.
The problem is that after running detectCoaddSources.py on a coadd, the mask in the resulting calexp image is gone (all set to 0), even for a coadd created by the pipeline itself. Here is how I run it:
detectCoaddSources.py $ROOTDIR --rerun=$RERUN --id tract=$TRACT patch=$PATCH filter=$FILTER
The only code I see to clear masks as part of detectCoaddSources.py is in SourceDetectionTask, and that only clears the DETECTED and DETECTED_NEGATIVE planes.
To run detectCoaddSources.py stand-alone, you should be creating a deepCoadd dataset to feed to it (not deepCoadd_calexp, which is an output of detectCoaddSources.py). Is that what you’re doing?
Something weird is that when I use the afw Display tools to look at the image, the mask appears fine. It is only when opening the image HDU #2 with ds9 (the latest version) that the mask is zero everywhere.
(I do run detectCoaddSources.py from deepCoadd).
Ah, that’s a known bug in ds9 support for the kind of compression we’re now using by default on our images. I believe it’s fixed in the very latest version of ds9 (but most testing has been done on OS X; I don’t know if we ever confirmed that the latest ds9 works on Linux).
It might be related but I don’t think it’s the explanation because 1) I do use the latest version (as I wrote previously) and 2) when I open the corresponding calexp exposure that was made by the pipeline (I assume from the run of coaddriver.py) I see the mask with this version of ds9.
But, try it yourself, run detectCoaddSources.py on a x,x.fits image and open the mask with ds9 on a system you’re sure it’s working with calexp images
Ok, that is quite strange. When the mask appeared successfully with afwDisplay, was that before your code wrote it out, or was it after writing it out and reading it back in?
In fact the issue is not linked at all to my emulate code. The problem occurs even if I use a pipeline-produced fits image and run detectCoaddSources.py on it.
It seems I can’t even get a ds9 (7.6, built for Ubuntu 17, running on Ubuntu 18.04) that will display the mask of a deepCoadd_calexp written by the main pipeline, if loaded directly via ds9 -multiframe, and I’ve confirmed that the image I’m testing does have nonzero mask values if I load it with afw. I can, however, use the same version of ds9 to read the mask of a non-coadd calexp (i.e. from the CORR directory)
That’s frustrating, but it’s also pretty clear that it’s a ds9 bug, not one of ours, though I don’t know why one kind of image works while the other doesn’t (maybe @price does?). What’s more confusing about your case is that I think you’re seeing both success and failure with different instances deepCoadd_calexp (unless you meant "non-coadd calexp" above) - is there any chance the original one that does work was produced by an older version of the pipeline?
There was a bug in cfitsio that affected reading compressed masks. Maybe that got into your ds9 build?
Can you try the latest beta release?
Hi both, three things:
detectCoaddSources.py with the same version of the pipeline that created the “working” (with ds9) deepCoadd_calexp, hscPipe 6.5.1. -> Same problem (ds9 does not show the mask)detectCoaddSources.py - deepCoadd_calexp with astropy and confirmed that the mask is working well and has non zero values (as afwDisplay tests suggested)The bottom line is: it’s indeed a ds9 bug, but not the one you suspected and that was already reported before (although probably related) , since I can read deepCoadd_calexp with the latest ds9 version.
It is still weird that it only happens when I run detectCoaddSources.py on my machine. The only test I can suggest to you is to try on your machines first to confirm, and then we file a bug report to ds9 developers.
I’m a bit late to this topic, but I was having this bug on my mac using ds9 version 7.6b4. The problem was fixed for me when I upgraded ds9 to v7.6