The DM monthly status report covering May activities has been posted to DocuShare, collection-1101. For convenience, the High-level Summary is pasted below. Direct link to the full report: https://docushare.lsst.org/docushare/dsweb/Get/report-1122/DM%20Monthly%20Progress%20Report%20202006.pdf
High-level Summary
Community Interactions, Meetings and Workshops
Several members of DM attended the LSST Solar System Readiness (Virtual) Sprint 3. Subsystem Scientist Leanne Guy spoke about Rubin Observatory Year 1 Template Generation and Year 1 Operations, while Mario Jurić, Siegfried Eggl, and Nate Lust (all members of the Science Pipelines teams) presented on Small Bodies with the Rubin Observatory and the Solar System Processing Pipeline and gave a tutorial on the Simulated Solar System Data Products Catalog. Video recordings of all of these talks are available online.
Clare Saunders and Meredith Rawls from DM both participated in the Satellite Constellations 1 (“SATCON1”) workshop. Clare presented her work on mitigating streaks in in Rubin images, while Meredith discussed observations of SpaceX’s “DarkSat”.
This month also saw a Pre-Verification Review of Rubin Observatory Networks. The committee commended anticipated performance and resiliency of the international network design, its use of state-of-the-art facilities, and the knowledge and experience of the team that is assembling and operating it. However, they made a number of useful recommendations which will be incorporated into our future planning. The report is available as Document-359345 on Docushare.
The first DRP metrics meeting took place. This is the start of a new effort to bring together members of the DRP team with their peers in Science Validation and Commissioning to evaluate the current performance of data release pipelines and to share techniques and best practices for pipeline verification.
Throughout this month, the DM team has worked to prepare documents and presentations for the Joint Director’s Review, which will take place in July.
Technical Progress
June saw the release of version 20.0.0 of the Science Pipelines. This major release addresses some 402 separate Jira tickets, and adds a number of new capabilities which are described in detail in the release notes. A characterization metric report, DMTR-251, accompanies this release, and an associated acceptance test campaign is currently being conducted by the DM System Science Team.
June also saw the submission of change request LCR-2376, which formalizes changes to our baseline plan for solar system processing. These changes, which have been discussed in several previous reports, notably call for closer cooperation between Rubin and the Minor Planet Center. We believe that ultimately they will lead to a more robust and scientifically useful system.
DM’s documentation infrastructure was moved to GitHub Actions this month, considerably reducing user wait times for new document builds and dramatically improving the user experience for DM staff.
The SQuaRE team introduced “Mobu”, a user simulator and testing service for the Science Platform. This already proved its worth, as it was used to investigate and fix a rare bug in the notebook service which only manifested under high load.
An extension to the XRootD distributed messaging system which has resulted in improved query load balancing across Qserv shard servers was implemented.
At the NCSA Data Facility, commissioning a second Qserv cluster got underway. Meanwhile, we demonstrated small-scale Data Release Production execution on cloud facilities using Google Compute Engine.
RFC-701, which proposed replacing Oracle with PostgreSQL in the consolidated database, was accepted, and this transition is now underway. We believe that PostgreSQL will be able to meet Rubin’s scalability and reliability requirements, while reducing the development overheads imposed on the development team by database compatibility concerns.
The alert packets generated by our pipelines were enhanced, including an improved schema, use of Astropy CCDData to encapsulate image data, information on DIAForcedSources, and cut-outs from template images. Finally, we completed a complete execution of the Alert Production pipeline using the new Generation 3 middleware, which marks a major step towards being able to adopt this middleware system-wide.