The DM monthly status report covering May activities has been posted to DocuShare, collection-1101. For convenience, the High-level Summary is pasted below. Direct link to the full report: https://docushare.lsst.org/docushare/dsweb/Get/report-1151/DM%20Monthly%20Progress%20Report%20202007.pdf
High-level Summary
Community Interactions, Meetings and Workshops
DM prepared for and participated in the Rubin Observatory Joint Director’s Review during the week of July 20. The DM subcommittee answered ”yes” to all relevant charge questions, and the subsystem received four helpful recommendations for future work.
Several members of the subsystem attended the Dark Energy Science Collaboration (DESC) meeting this month. Pipelines Scientist Robert Lupton and Subsystem Scientist Leanne Guy presented the current status of the Science Pipelines, while Data Release Production Science Lead Jim Bosch presented progress on the new “Generation 3” middleware. Bosch also gave a presentation to the Strong Lensing Science Collaboration.
Nate Lust, a member of the DRP team, attended PyHEP 2020 at which he gave a presentation entitled Rubin Observatory: the Software Behind the Science.
Technical Progress
DM organized and, in conjunction with other members of the team, executed the second Operations Rehearsal during the week of 27 July. This three-day event saw data transferred from ComCam in the La Serena Base Facility to the Data Facility at NCSA, where it underwent pipeline processing and QA checking. This successful activity clarified a number of important operational considerations, and there is enthusiasm across the project to press forward with further similar rehearsals. A full report on this rehearsal will be issued as DMTN-159 during August.
A selection of sample alert packets, and a system to deliver them in a simulated alert stream, were made available to the community. This is expected to be especially useful to teams working on proposals for community alert brokers (LDM-612), but we hope it will be of value to the wider scientific community interested in low-latency science with Rubin.
A number of important improvements were made to the Science Pipelines this month. Highlights include the completion of a full density simulation of observations of solar system objects during year one of the LSST survey; the capability to apply proper motions and parallaxes to reference sources in the Jointcal astrometric fitting system; and a dramatic improvement in I/O performance in the Forward Global Calibration system. Further SCARLET — the prototype deblender algorithm — now has memory usage patterns which are comparable to the default deblending algorithm on precursor data. This means we are now able to proceed with large-scale testing of this algorithm.
We continue to work on a proof-of-concept of processing in the Google Cloud. This month, we demonstrated acceptable levels of capability for Data Release Processing and prompt data transfer, with further improvements expected in future.
Two landmarks were reached for the Qserv distributed database system this month: the DESC Data Challenge 2 catalog was successfully ingested into a Qserv instance at the CC-IN2P3 French Data Facility, while a second Qserv cluster was commissioned at the NCSA Data Facility specifically to support ongoing operation of the Rubin Science Platform.
Also at NCSA, successful execution of the “ci_hsc” test pipeline based on the Generation 3 middleware, using a PostgreSQL backed registry, and using the ctrl_bps Batch Production Service was demonstrated. This illustrates major progress made in upgrading systems across DM to take advantage of the capabilities provided by the new middleware and execution systems.
Networking tests between La Serena and NCSA continued this month. In particular, we note the upgrade of the La Serena to Santiago secondary link to 40 GBit/s.
This month saw a major upgrade to our documentation system, as the SQuaRE rolled out an improved documentation hub at https://www.lsst.io/. This provides much improved searching and indexing of a range of project technical and software documentation. In addition, updates were made to a number of documents. In particular, we highlight:
DMTN-156, which presents performance testing of a three-node Cassandra NoSQL database cluster as a potential APDB solution, and
SQR-044, which discusses requirements for the identity management system of the Science Platform and related services.
Revised drafts of a number of documents were issued in advance of this summer’s reviews. In particular, RFC-708 proposes updates to LDM-148, LDM-294, LDM-503, LDM-564, and LDM-643, which we expect to baseline before the Joint Status Review at the end of August.