The DM monthly status report covering September activities has been posted to DocuShare, Collection-6145. For your convenience, the High-level Summary is pasted below. Direct link to full report (pdf): http://ls.st/plq.
High-level Summary
The DM Project Manager and Subsystem Scientist attended the PST meeting at SLAC to discuss potential adjustments to DM scope, with a particular focus on pausing Firefly development, and to present the approach being taken to acceptance testing and verification. In related news, the Architecture Team released the first version of “Docsteady”, the tool which will be used to automatically generate verification documents.
The System Science Team made significant steps forward in relations with the science community, as they issued the first version of the DM Glossary and set up a roster of SST members who will attend and participate in the “Stack Club” sessions which introduce science collaboration members to the LSST software stack.
The Data Access and Database Team and the LSST Data Facility continued work with Google on proof-of-concept deployment of the Qserv database system within Google’s GKE environment. As part of this effort, WISE catalog data was successfully transferred from the prototype data access center (PDAC) to the Google “cloud”, where it will used as the basis of performance comparisons. Refinements necessary to the Qserv Kubernetes deployment tooling in order to facilitate testing within GKE have been identified, and implementations are now underway.
Science Platform development continued apace. The Firefly tool gained the ability to smoothly transition between 2D histograms for large datasets and simple scatter plots when the data volume is reduced by filtering. A prototype of a JupyterLab extension was development which enables it to display FITS images in a tab using Firefly. Finally, Dask, a system for parallel computing using Python, has been deployed next to our notebook service to provide users with the capability to run computations over large data volumes: the prototype was tested with the 1.7 billion row Gaia DR2 catalog.
The Alert Production pipeline was enhanced with the ability to perform forced photometry on processed visit images, while the Data Release Production system gained new capabilities for training its machine-learning based star/galaxy classification capability. A new framework for LSST reference catalogs, which includes support for proper motions and parallaxes, was rolled out across both Prompt and Data Release processing systems. We are delighted to welcome Siegfried Eggl and Gábor Kovács to the DM team. They will spearhead development on moving object processing and image differencing, respectively — two of the highest-priority areas of algorithmic work over the coming years.
A breakthrough was achieved by the members of the Data Release Production, Architecture, and Data Access teams working on the “next generation” middleware which will be used to execute those pipelines, as the first pipeline “task” converted to use the new middleware was successfully executed and produced outputs.
Improvements have been made to the L1 forwarder/archiver system and Header Service on the Auxiliary Telescope test system in Tucson, resulting in the ability to successfully collect images over an extended period.
Summit users have been switched off the temporary network to the core network and DWDM on a trial basis. The fibers connecting Calibration Hill to the Summit Computer Room were installed in support of Auxiliary Telescope DAQ and CCS testing in November and of Auxiliary Telescope observing in early 2019. The Santiago ring and La Serena – Santiago primary and backup links are fully operational. We received verbal agreement from DOE, OHEP and ASCR to move ahead with implementation of the ESnet link from ATL to CHI.