Verification Datasets Meeting 2016-02-17 Minutes

Data Management
#1

Verification Datasets meeting
Attendees: Mario, Zeljko, MWV, Colin, Hsin-Fang, Swinbank, RHL, Simon
Notetaker: David

Colin:

  • rescale noise/variance plane
  • what’s the status of the noise plane, Colin won’t persue this, Frank could look at tech notes
  • correlated noise, some features that look like crosstalk issues, but still needs to investigate more

Mario:

  • Joachim has MOPS things running

  • compilation on linux still an issue

  • work through problematic linkages

  • false positives, no KPM, a metric in OSS, want less than 1000 deg^2 to get MOPS running
    are happy with this in the short term for NASA and NEOs, but not for the long term

  • David: multi-threaded, multi-process?

  • Mario: MOPS has lots of knobs, need to understand it first,
    can process 1 month of simulated LSST observations on laptop in a few days, independent linkages for 2 week blocks, should parallalize well, not that computationally challenging

  • Colin: noise floor is ~34 per deg^2, we are 10x that now, but some could be real

  • Mario: talking to JPL counterparts on Thursday, generating catalogs for them soon

Zeljko:

  • no update, just getting ready for JTM

MWV:

  • working on tech note, http://dmtn-008.lsst.io/

  • validate_drp, adding capabilities

  • output kpm metrics to file

  • David should review multifilter matching capability
    PA1, PA2, photometry repeatability
    AM1, AM2, astrometric repeatability

  • run on CFHT and DECam COSMOS on validation data
    runs fine, DECam CP results are not good

  • looking at ci_hsc using scons

  • Frossie: are people capturing their verification work in JIRA? Colin, Hsin-Fang?
    Hsin-Fang sends things to Margaret,
    Colin hasn’t been tracking things in JIRA, Mario has been sending reports to Jeff
    Frossie encourages people to track things in JIRA the way MWV has been doing

  • MWV: two exposures are good enough to measure the KPMs, can do larger exposure too.

Hsin-Fang:

  • working on other stuff last week
  • figure out what’s next on her place at JTM

Robert:

  • nothing to update
  • waiting for data from DECam calibration run

Angelo: (not present, brief update given by David)

  • working on QA database and visualization
  • LIneA work

David:
https://confluence.lsstcorp.org/display/SQRE/COSMOS+DECam+data+reduction

  • doing a bunch of SMASH work, reduction, matching and calibrating data
  • updating script to measure and collate metrics for a data repository
  • CR issues, correlate strongly with background level, many CRs not being detected/masked
  • RHL: straight ones - muon, worms - electrons
  • program to create subtracted image, RHL has code to do this
  • RHL surprised at how bad this is, much better on HSC side
  • shouldn’t have to tune any config parameters
  • seeing looks fine
  • RHL and David will work on figuring out what’s going on with CR code on this data

Simon:

  • twinkles, starting to run input catalogs, have that done by tomorrow

  • for March meeting, single-simulated central chip, deep drilling, all bands, once per night for entire survey, 1800 images, using OpSim, lowest airmass observation per band every night

  • variability built in, SNe, 100 5 sigma detections per visit, standard AGN variability

  • some implanted multi images for lensed galaxies,

  • this chip happens to not have any RR Lyrae

  • have simulated images done by March

  • turned off airglow variations and clouds

  • saturation, CRs are on, B/F off, nonlinearity turned off, no ghosts

  • semi-idealized situation

  • MWV: what outputs are you planning on analyzing?

  • Simon: no firm plans, hope the Twinkles team should help figure out what to do with the outputs

  • Simon will ask Andy C. about the DM simulated data needs

  • processCcd rewrite update

  • nothing will be merged with a regression

  • probably a catalog problem

  • matcher is fragile

  • works fine in DECam, HSC, has some issues on CFHT

  • MWV:

  1. expect some differences when doing large refactoring
  2. take advantage of Dominique’s
  3. be useful to have list of tests they are doing

  • Simon: make these debugging/QA acitivities “first class” activities that are scheduled

  • Frossie: please capture this work

  • RHL: use debug options, for post-mortem use ipython notebooks

  • Colin: where should ipython notebooks go?

  • Frossie: have an answer by next meeting

  • Colin/RHL: currently putting ipython notebooks on personal github account

  • RHL: are we using the same stars for the matching, different matching radius

  • deblending

  • MWV, could deblending be causing problems

  • RHL, normally deblending should improve things