The DM monthly status report covering May activities has been posted in DocuShare, Collection-6145. For your convenience, the High-level Summary is pasted below. Direct link to full report (pdf): http://ls.st/s5j.
High-level Summary
The Data Management Leadership Team convened in Seattle during the week of 21 May for a three-day face-to-face meeting. This meeting addressed many important topics, with highlights including reports on the progress of DM working groups; a status update on Summit–Base ITC and networking; discussion of pipeline output data formats; future approaches to managing the DM codebase; evolution of the DM product tree and approach to verification and validation; and planning for the Fall 2018 development cycle, which starts in June. The DMLT meeting was preceded by a DM System Science Team meeting, which focused on developing verifiable numerical requirements on various aspects of the DM system’s performance.
Other major meetings with DM involvement this month included:
- The LSST Commissioning “Bootcamp”, which took place in Tucson, provided an opportunity for members of the Commissioning Team to become familiar with the Jupyter notebook environment being provided by DM and to clarify how DM will interact with the commissioning effort;
- The Canadian Astronomical Society meeting in Victoria, BC, at which Melissa Graham presented the LSST data products and information on participation information;
- The IVOA Interoperability Meeting, also in Victoria, BC, at which several members of the DM team presented updated on LSST’s plans for data access technologies, alert distribution, and integration with the Virtual Observatory, as well as demonstrating new capabilities — including all-sky visualization — in the Firefly tool. At this meeting, DM team member Tim Jenness was appointed to the IVOA Executive Committee, and Gregory Dubois-Felsmann accepted a position on the IVOA’s Committee on Science Priorities;
- The DESC photometric calibration workshop, in Paris, France.
In addition to these activities, there were many important advances in DM construction. Prominent amongst these are the new DM product tree and test management system, which have been developed by the DM System Architecture team (the latter in conjunction with Project-level Systems Engineering).
Development continues apace on the new “Generation 3” data access middleware (”Butler”). This continues to be a major cross-team effort, with key contributions from the Systems Architecture team, the Science Pipelines group, the LSST Data Facility, and the Data Access team.
The scientific algorithms being developed by DM also continue to advance. This month saw the prototype Alert Generation Pipeline calculating and storing its first aggregated quantities for DIAObjects, and key upgrades to the system used to select which images are used to generate templates for image differencing, and an update to the Forward Global Calibration Method to enable it to produce high-quality atmospheric transmission functions.
An important milestone in the development of the LSST Science Platform was realised as the SQuaRE team successfully incorporated the Firefly visualization tool, developed by the SUIT group, into their Jupyter notebook system. At the same time, we upgraded the Prototype Data Access Center (PDAC) to provide asynchronous TAP query support.
The LSST Data Facility completed initial configuration of the core Kubernetes cluster — a key piece of DM infrastructure — and made it available to users. They also completed provisioning and configuration of the ATS turnkey system, and shipped it to Tucson. The system will be used to interface the Auxiliary Telescope and DAQ to the L1 Archiver System and Header Service.
The Networks group started preparations for a CY18 100 gigabits/second demonstration in conjunction with Supercomputing 2018. We received the ESNet for LSST cost analysis. The costs came in quite close to the original estimate, and we are updating the LSST Operations Proposal costs to reflect this.
This month also saw the inauguration of the DM “Quality Assurance Working Group”. This group, chartered in LDM-622, will produce a report and recommendations on the plans and requirements for ensuring that developers and the project as a whole have access to the tools and capabilities required to effectively develop and debug scientific algorithms at the scale of LSST data releases. It is due to report by the end of June.
Finally, preparations continued throughout the month for the Directors Review, in June, and Joint Status Review, in July.