The DM monthly status report covering March’s activities has been posted to DocuShare, collection-843. For convenience, the High-level Summary is pasted below. Direct link to the full report (pdf): http://ls.st/4mr
High-level Summary
Community Interactions, Meetings and Workshops
Several members of the DM team attended the Dark Energy Science Collaboration meeting from July 15 to 19 in Paris, France; some also participated in the DESC Photometric Calibration Working Group meeting the week before. This meeting provided a key opportunity for members of the DM team to interface with a key science collaboration, and forms the background for a great deal of valuable technical discussion.
In addition to the DESC meeting, members of the DM team attended the SciPy 2019 and Python in Astronomy 2019 conferences during July. Both meetings provided opportunities to learn about best practice in software development for astronomy and to showcase some of the innovative technical work that LSST is carrying out.
Management effort focused on preparations for the Joint Directors’ Review (JDR) and Joint Status Review (JSR) which will take place in August. This included revising and updating baseline documents and shepherding them through subsystem and/or project-level change control. A dry run for the reviews took place July 22–24, and went smoothly.
The DM System Science Team presented a study on options for alert production in the first year of operations to the Project Science Team. In later years of the survey, alert production will be based on differencing science images against deep templates drawn from earlier data releases; obviously, this is not possible before a data release has taken place. The SST study has identified several options which are now under discussion by the PST.
Members of the DM team participated in a two-week “puppeton” in Tucson, which included presence and/or telepresence from multiple subsystems. The aim of this meeting was to push ahead with Puppet/Foreman-based deployment of IT systems in Tucson, and Puppet configuration of systems provided by NCSA.
Technical Progress
Several key DM baseline documents were updated in advance of the project reviews which will take place in August. These include:
LDM-148, the DM System Architecture
LDM-152, the DM Middleware Design
LDM-294, the DM Management Plan
LDM-672, the DM Software Release Management specification.
In addition, several important (but non-baselined) technical notes were issued, including:
DMTN-112, describing the impact of variable seeing on DCR mitigation
DMTN-122, the Data Backbone Design
DMTN-123, the Batch Production Service Design
DMTN-125, reporting on the Google Cloud proof-of-concept
SQR-032, describing plans for rendering and testing examples and tutorials in LSST documentation
SQR-033, describing plans for future development of the SQuaSH service in light of the recommendations of the QA Working Group (DMTN-085).
RFC-620 was filed, proposing a new approach to solar system object processing, as described in DMTN-087. This proposal is currently under discussion within the subsystem; assuming it is agreed upon, a project-level change request will be submitted to capture the new plan.
Release 18.0.0 of the Science Pipelines was issued. This major release contains a number of new features and improvements; refer to the release notes for details.
The Engineering & Facility Database (EFD) architecture proposed by the DM team, based on the Apache Kafka streaming platform and InfluxDB time-series database, will be adopted as the facility-wide system. Consequently, we expect work on this to be prioritized over the coming weeks.
Major progress was made in the development of the new “Generation 3” DM middleware as we demonstrated successful processing of a “tract” of Hyper Suprime-Cam data through a data release pipeline using the new middleware, backed by an Oracle database backend, on the Batch Production Service.
Science Pipelines performance was substantially improved by changes and optimizations made to AssociationTask (which associates DIASources, corresponding to detections on image differences, to form DIAObjects) and galaxy fitting in the MultiProFit tool.
A significant piece of rework in the Qserv query-processing front-end was completed, leading to resolution of several outstanding issues, improving query coverage in the TAP query service layered above, and paving the way for easier future code maintenance.
NCSA finished pre-shipment configuration and testing of systems for Base Data Center database services, storage, data transfer, Auxiliary Telescope System, and Engineering and Facilities Database, and shipped them to Chile for installation in August.
Work continued on the campus and control wired and Wi-Fi networks to all areas, floors, and rooms of the Summit Facility, the Auxiliary Telescope and auxiliary instruments, and the Base Data Center and offices. The Continental United States (CONUS) Network Implementation Team (CNIT) was formed and is led by Paul Wefel of ESnet.