Verification Datasets Meeting 2016-07-13 Minutes

Verification Datasets telecon
Attendees: Simon, David, MWV
Regrets: Colin


  • trying to make a repository that is agnostic to camera
  • put files in repository and run processCcd on it (must have previously ISRed)
  • just need camera info for ISR
  • need to know variance pretty well otherwise CR rejection falls apart
    if background subtracted then you need to give the noise level (scalar)
  • just assumes a single 2D grid of flux values, no variance or mask plane, but could add that in
  • giving people the ability to use processCcd like SExtractor
  • it internally bulids the variance (using gain, config parameter) and mask plane
    can also set lower threshold for “good” values, also needs to know the noise level
  • uses repository, can use Butler to get the source catalog, etc.
  • could actually hide the butler by adding a little bit of code that checks if the
    file exists in the repository and if not then it adds it
  • currently a branch under SimonKrughoff obs_file
    tickets/DM-6924 branch of both obs_file and pipe_tasks
    c.l.o post about this
  • probably useful for the collaborations


  • had meeting with PanSTARRS people at MPIA and discussed QA, mostly done after the fact to see what issues there are (forensic mode)

  • Simon, discussion of prompt processing, send out alerts on 99.9% of images that aren’t corrupted by any of the processing

  • image also needs to be scientifically useable, who makes this choice

  • maybe use OCS flags?

  • need red light/green light on Level 1 products

  • NCSA is putting up a verification cluster and they wanted to know what workflow people want,
    slurm and condor a possibility

  • SQuaRE doesn’t want to much effort on developing a new workflow system

  • David, want to run PSFEx on COSMOS to see if the PSF model and photometry is better

  • CR, problems on DECam COSMOS images, but SK ran some it found ~80% of the CRs

  • is CRblaster work better, CP uses it to identify and mask them
    should compare LSST to CRblaster

  • MWV suggested having a CI test to find a certain number of CRs in an image

  • issues with variance plane can cause problems with the CR software, was an issue with
    CP-processed DECam data in the past, wanted to double-check with LSST ISR but never quite got there


  • planning improvement for LDM-151 on SDQA
  • a lot of pieces in SDQA still need to be designed

CRblaster uses the LACosmic algorithm, which is simply Laplacian edge detection. I believe our CR code does essentially the same thing. It may just need a front-end that correctly estimates the variance and PSF, and a bit of parameter tweaking (e.g., the background subtraction scale). I also found on HSC that setting cond3_fac2=0.4 helps.