DM Monthly Status Report for June

Data Management DM Notifications
#1

The DM monthly status report covering June has been posted in Docushare, Collection-4640. For your convenience I am pasting high-level summary and project status below. Direct link to the full report (pdf): http://ls.st/o2i

High-level Summary

The DM Team released version 12 of the LSST DM Software Stack, bringing improvements to
linearity correction, precursor camera support, PSF estimation, coaddition, and new shape
measurement algorithms (among others). Computing hardware continues to arrive and is being
integrated at NCSA. Additional layer of algorithmic detail was added to LDM­151 (Science
Pipelines Design Document), and project management processes and tools documents were
completed and submitted to TCT for approval. The team continues refining and executing the
construction plan, and is actively preparing for the upcoming reviews.

Project Status

The DMLT continued preparing for the upcoming reviews, and started planning the DM portion
of the LSST2016 meeting.

We made a thorough revision of LDM­151 (DM Science Pipeline Design Doc), and completed
documenting our project management processes and tools (LDM­PMT), as well as the technical
note describing our project management best practices (DMTN­020).

We completed analyzing X16 cycle, and transitioned to the Fall 2016 (F16) development cycle.
Based on the analysis, the team completed 88% of planned X16 activities and some activities
from the upcoming F16 cycle, the shuffle was primarily driven by the Working Groups needs
and adjusting priorities; the effective team velocity during the X16 cycle has been 99.1% of the
planned velocity.

A few members of the team attended the SPIE conference in Edinburgh, the LSST@Europe2 in
Belgrade and ASKAP in Sydney. We also participated in the LSST Camera Workshop at SLAC.

Selected technical accomplishments:

  • The Architecture Team with participation from NCSA had extensive discussions at the
    Camera Workshop about refinements and modifications to the LSST system design,
    including providing services at the summit and ensuring adequate scriptability of the
    system during commissioning. NCSA, along with the team also began defining the scope
    and deployment architecture for the Prototype DAC to be set up later this year, and
    prepared to enable the DM System to run under Python 3.
  • The AP Team implemented a real space implementation of a image differencing noise
    de­correlation algorithm, and a prototype DCR template correction algorithm. The team
    also added generalized linearity correction to ISR
  • The DRP Team added a first version of a package version tracking (“provenance”)
    system to the Science Pipelines, and added the capability to perform Gaussian mixture
    model based measurements using the ngmix external package to the LSST stack.
  • The SUIT Team finished the Firefly API in JavaScript in the new JS/React/Redux based
    system. The API is in integration and test phase with IPAC/IRSA applications. The team
    combed through the SUIT requirements and will start rewrite of the requirement to make
    it in line with SUIT vision.
  • The DAX Team continued to investigate, tune, and improve Qserv shared­scan
    performance on the IN2P3 cluster, developed a script that can provision an ad hoc
    multi­node Qserv cluster on OpenStack ab initio, and then perform integration tests on
    that cluster. The team began work to containerize DAX web services for deployment in
    the upcoming prototype DAC.
  • In the area of networking, on June 22nd, the second 100G link between South America
    and Miami was installed. This link currently connects Miami to Sao Paulo, and it will be
    opened in Chile in October/November. To evaluate the link, AmLight is generating 80
    Gbps of traffic using the 100G IXIA Tester installed in Miami. The 4G link to Santiago
    from La Serena has been carrying test traffic for a month now with no problems. The
    problem with the Perfsonar has disappeared and is working perfectly.
  • The SQuaRE Team released version 12.0 of the LSST DM Stack, completed the status
    report for DM Verification, and added Python3 support to the CI system.
  • At NCSA a very substantial installation activities occurred as materials for the FY2016
    hardware procurement arrived. The Data system replanning effort produces concepts of
    operations needed to further plan the delivery of of working services as a result of
    construction.