Looking at a flat of the same detector, we find a real defect at the x0 location specified by the defects mask (orange circles represent corners of the masked region) but spanning one or a few rows instead of a large region:
I am wondering what the history/status of this defects mask is, as I can’t find reference to it elsewhere, say at Status of DECam CCDs | NOIRLab Science. Could this region have been masked as a box mistakenly when it should be a partial row?
Hi @stevenstetzler – thanks for the question, and for digging quite deeply into this already.
It appears to me that the most recent defects file for DECam CCD N26 is from January 2015 (almost 10 years ago!), so I wouldn’t be surprised if something about DECam CCD performance or understanding of how to calibrate DECam data e.g., for certain CCDs has changed since then. In addition to being old, I believe the relevant defects files were inherited from DES, so questions about their history are probably better addressed to that collaboration than to Rubin/LSST. Setting that aside, are you able to customize the relevant ECSV file within $OBS_DECAM_DATA_DIR/decam/defects of your LSST Pipelines installation? That would seem like a fine solution to me.
Regarding that NOIRLab webpage, I would not necessarily expect it to track closely with LSST Pipelines development — NOIRLab has a different DECam data reduction pipeline (the Community Pipeline), which I would expect to have a series of versions more closely correlating with updates listed on that NOIRLab page.
Also, I believe that at least a couple of the top obs_decam contributors (@parejkoj and @mrawls) are at UW, so you might consider chatting with them as well. Thanks again.
Hi @ameisner, thanks for your help! I just chatted with @parejkoj and he suggested @lskelvin might know more or would at least want to know about this. I think the defects were inherited from DES, so yeah the answer might lie in that realm instead of in this group. I think modifying the ECSV would work for bespoke processing, and if the history of the defects mask is know and it is a confirmed a mistake the modification could be merged into a future release of the pipelines. Knowing what the right mask is seems out of scope for any of us to figure out though unless @lskelvin has worked on this before or we can get an updated mask from DES. @lskelvin do you know anything about this? Or would it be best to ask around in DES for an answer?
Having the right mask is not the highest priority for us right now anyway, but this seemed worth asking about.
Hi @stevenstetzler – I’m not sure who the right person would be to ask within DES, but on the NOIRLab side I’d suggest emailing Frank Valdes, the author of the DECam Community Pipeline software, and ask him whether there’s ever been any major known issue with that big rectangular region of N26. If you’d like I could be the one to email Frank and copy you on that.
It’s hard for me to speculate as I’ve never been part of DES, but it’s a precision cosmology project so for all we know that detector area was flagged so as to be conservative about a subtle effect, rather than to get rid of catastrophically bad data. This also relates to the point that the optimal mask could be a function of the science application, in which case different groups customizing their defect files may be inevitable. Thanks again…
Hi @stevenstetzler, I’ve been following this thread with great interest. We indeed are using DECam defects definitions that have their origins from a number of years ago. If there are updates to be made, I think we should aim to merge these into DECam main to support future DECam data reduction efforts with the Science Pipelines. @ameisner, if you are able to find our more information from Frank or others within DES, that would be fantastic, thank you.
Thanks @lskelvin. @ameisner I’ve reached out to Frank and a collaborator has reached out to folks in DES about this. I’ll provide an update as we get more information.
The result of initial sleuthing is that the DECam CP does mask out the row in question as BAD (see the bpm from DECam Calibration Files | NOIRLab Science) and that the DES folks likely interpret that row as BAD and the pixels that follow that row in readout as SUSPECT, not to be used in precision measurements . However the LSST Science Pipelines are applying a BAD mask to the entire region, causing masking and interpolation over that region e.g. in the top right portion of this difference image:
On GitHub - lsst/obs_decam_data: Repository to hold data related to the obs_decam package. it indicates the current defect mask is compiled from datasets/decam/_internal/calib/bpmDes which I’ve tracked down on the USDF at /sdf/group/rubin/datasets/decam/_internal/calib/bpmDes. I took a look at the mask for detector 57 (/sdf/group/rubin/datasets/decam/_internal/calib/bpmDes/2014-12-05/D_n20150105t0115_c57_r2134p01_bpm.fits) and indeed the mask has value 512 over the region of interest which corresponds to BPMDEF_SUSPECT per DES Image Processing Table 8:
My best guess based on this is that the pipelines defects were generated by taking the pixel locations with non-zero values in the bpmDes images. This included the SUSPECT pixels which aren’t necessarily bad, but may mask out large regions like what is seen in detector 57.
@lskelvin if this seems right to you and we don’t want to mask the SUSPECT pixels as bad, it might make sense to rebuild the defect masks from the bpmDes directory excluding the SUSPECT pixels. And/or include information from the existing community pipeline bad pixel masks. What do you think?
My best guess based on this is that the pipelines defects were generated by taking the pixel locations with non-zero values in the bpmDes images. This included the SUSPECT pixels which aren’t necessarily bad, but may mask out large regions like what is seen in detector 57.
The obs_decam_data repo’s README seems to say that this is indeed what happened?
These masks are turned in to LSST style mask images with the single BAD bad bit set. Any non-zero pixel in the BPM was assigned the BAD flag.
Hi @stevenstetzler , as Aaron said, thanks for the deep dive on this. @lskelvin , do you have any thoughts on Steven’s latest post and question?
@lskelvin if this seems right to you and we don’t want to mask the SUSPECT pixels as bad, it might make sense to rebuild the defect masks from the bpmDes directory excluding the SUSPECT pixels. And/or include information from the existing community pipeline bad pixel masks. What do you think?
Thanks for doing a deep dive here @stevenstetzler. On the face of it, your logic seems sound. If we indeed want to use these pixels which have been identified as SUSPECT, then a modification of the defects file inside the Science Pipelines can be made. Before committing to that though, I would appreciate it if you were able to test out any required edits on a forked local version, to ensure that downstream processing does not suffer when using a modified defects mask. If all proceeds ahead, we can set up a PR and have this merged into the main code base.
