The DM monthly status report covering February activities has been posted to DocuShare, collection-1022. For convenience, the High-level Summary is pasted below. Direct link to the full report https://ls.st/k2e
High-level Summary
Community Interactions, Meetings and Workshops
The Data Management team held a very successful all-virtual Rubin Observatory Algorithms Workshop. The decision was taken to move the workshop entirely online in light of the evolving COVID-19 situation. In practice, this worked well, and the team learned a lot about facilitating remote meetings. All talks were recorded, and are now available from the workshop website at https://ls.st/law.
Members of the Data Facility team from NCSA attended the Third Rucio Community Workshop at Fermi National Accelerator Laboratory. This included opportunities for in-depth discussion with Rucio developers, which were valuable in advancing our plans for the Data Backbone.
Technical Progress
Key reports and documents updated or published include:
DMTR-201, fall 2019 Pipelines Release Acceptance Test Campaign Test Plan and Report, was completed.
DMTN-143, describing a simplified raw image transfer design, was published, and a change request filed requesting that this design be accepted by the Project Change Control Board.
A partial Science Platform, including our JupyterLab notebook service, was deployed at the Cerro Pachón summit allowing commissioning staff to use the features of the platform in the same network as the hardware devices. The latency in the ingest of telemetry into Engineering Facilities Database was also significantly reduced. The new data transfer system — taking images from the LATISS instrument mounted on the Auxiliary Telescope and immediately transferring them to the Data Facility at NCSA — was placed into production.
The database team continued to exercise and refine the parallelized data ingest system for the Qserv distributed database. Gaia Data Release 2 was re-partitioned and re-ingested at the Data Facility in its entirety (including both source and secondary tables). Testing of the Cassandra-based prototype of the Alert Production Database continued at the Data Facility, with a focus on performance monitoring and tuning.
The Architecture Team took on official responsibility for the jobs running in the continuous integration system including the underlying build system, improved its performance, shrank the storage required by the containers it creates, and improved the handling of different Conda environments.
The Science Pipelines group deployed initial processing pipelines for DECam data based on the “Generation 3” middleware. Python implementations of code for initial orbit determination (IOD) and moving object processing (MOPS) were also made available.
A new algorithm for detecting and masking satellite trails was developed and is now being incorporated into the codebase. A simplified and more robust system for correcting amplifier nonlinearity was deployed.
NFS file systems through the DDN GPFS appliance at the Base Data Center configured and set up. This provides about 1 PB of storage for the commissioning cluster and other subsystems as needed.
All systems at the Summit Facility, and the DWDM equipment, were shut down on March 20th as part of suspension of construction site activity due to COVID-19.
General Parallel File System
Dense Wavelength Division Multiplexing