The DM monthly status report covering May activities has been posted to DocuShare, collection-1101. For convenience, the High-level Summary is pasted below. Direct link to the full report: https://docushare.lsst.org/docushare/dsweb/Get/report-1164/DM%20Monthly%20Progress%20Report%20202008.pdf
High-level Summary
Community Interactions, Meetings and Workshops
Most members of the DM subsystem participated in the Rubin Observatory Project and Community Workshop, August 10–14. DM team members participated in the Scientific Organizing Committee for the meeting, and were involved in organizing and chairing several key sessions, including:
An Introduction to the Rubin Observatory: Systems, Jargon, and Acronyms
Rubin Algorithms Workshop Follow-up: Answers to Community Questions
Low Surface Brightness Science and Robust Sky Estimation
In addition, the DM team organized the now-traditional PCW board gaming session!
DM was represented at the DotDotAstro online conference on August 17, 19, and 26 by Meredith Rawls (University of Washington).
At the end of the month, members of the DM team were involved in the annual Joint Status Review. This started on August 31, and continues into September.
Technical Progress
Many documents were updated or published in August in preparation for the Joint Status Review. Notable highlights include:
LDM-148, the DM System Architecture
LDM-294, the DM Management Plan
LDM-503, the DM Test Plan
LDM-564, the DM Release Schedule
DMTN-157, documenting initial results of the Google Cloud proof-of-concept
SQR-048, providing security guidelines for Kubernetes clusters in use by DM
In addition, the documentation portal at www.lsst.io now has a tag-driven user guide listing in preparation for exposing the service more to external users.
Substantial progress was made towards adopting the new “Generation 3” data access and task execution middleware this month, as we deployed a complete “end-to-end” Alert Production pipeline using this new system. In addition, Hyper Suprime-Cam RC2 — one of our core precursor test and verification datasets — is now available in the Generation 3 system, with the data registry held in a PostgreSQL database at NCSA. Pipeline processing of this HSC RC2 data is now possible in the Generation 3 framework.
Several key refinements were made to DM’s data access and database systems during August. First, the WISE and Gaia Data Release 2 datasets were ingested and made available in a new Qserv deployment at NCSA. Secondly, the prototype Alert Production pipeline was upgraded to use a database based on PostgreSQL (rather than SQLite, as had been used previously); this provides much improved scalability and enables large scale testing. Finally, data aggregated (or otherwise processed in-stream) from Apache Kafka (such as the Engineering Facilities Database telemetry stream) can now be stored to S3 in Apache Parquet format.
A number of useful improvements were also made to our science algorithms. In particular, DM’s implementation of the ZOGY image differencing algorithm has been extended to support a spatially-varying PSF, and our task to inject simulated sources into data release processing can now accept arbitrary input images.
The generation of 21-raft images by the Camera subsystem at SLAC had major impacts across the whole Rubin team and beyond this month. DM’s involvement included transferring the data to NCSA, ingesting it into standard data repositories, and making it available for viewing and manipulation via Jupyter notebooks in the Science Platform.
The DM team is preparing for some major personnel changes in the upcoming weeks. Effective Oct. 1, Jeff Kantor will reduce his effort to 25% Rubin Obs. Cristian Silva, based in La Serena, will take over networks and co-chair the NET team. On a similar timescale, Deputy DM Project Manager and Science Pipelines Manager John will transition out of the project. His leadership role in the Science Pipelines group will be taken over by the new Science Pipelines Manager Yusra AlSayyad and Alert Production T/CAM Ian Sullivan.