The DM monthly status report covering August activities has been posted to DocuShare, collection-853. For convenience, the High-level Summary is pasted below. Direct link to the full report (pdf): http://ls.st/tzy
High-level Summary
Community Interactions, Meetings and Workshops
The Data Management team took part in both the Joint Directors’ Review and the Joint Status Review during August. Both reviews went well for DM, with reviewers acknowledging the hard work and substantial progress that has been made to date, and providing a number of useful recommendations and suggestions which will help us move forward effectively.
Many members of the DM Subsystem attended the LSST 2019 Project and Community Workshop. This was a productive meeting for DM, with a number of important sessions and key discussions taking place. Particular thanks are due to Melissa Graham, a member of the DM System Science Team, who chaired the Scientific Organizing Committee for the meeting, and to the several other members of the DM team who organized sessions or contributed presentations.
The manuscript “Local monitoring of atmospheric transparency from the NASA MERRA-2 global assimilation system,“ describing the work undertaken in DM to use the NASA MERRA-2 system to estimate atmospheric parameters, entered internal LSST project review late in August. Assuming the review is successful, we hope to submit it for journal publication in September.
Following the very successful Community Broker Workshop earlier this year, the DM team has compiled a “FAQ” answering many of the questions raised by potential broker teams. This is available on the LSST Community Forum.
DM has invested substantial effort over recent months revising its plans for processing crowded fields. Initial proposals were discussed with members of the Stars, Milky Way and Local Volume Science Collaboration during the LSST 2019 meeting, who responded positively. The DM team will now develop these proposals into a concrete plan.
Similarly, work is well underway on planning for changes to LSST’s solar system data products and processing system. These are being curated on RFC-620, and we took the opportunity of the LSST 2019 meeting to discuss them with the relevant science collaborations and other members of the community on them. We expect this work to culminate in a project-level change request in the near future.
Technical Progress
August saw the release of version 18.1.0 of the LSST Science Pipelines. This is a minor feature release, with some improvements in support of Hyper Suprime-Cam (HSC) data processing. Importantly, though, it demonstrates aspects of the new DM release procedure, and our ability to support making releases of this type upon request.
The astro_metadata_translator package, which is used to translate headers from diverse astronomical instrumentation into a standard form, was released to PyPI, the Python Package Index. This package is developed within DM, but is released separately from the rest of the LSST codebase to facilitate ease of installation and wide community adoption.
The Science Pipelines team deployed a version of the Forward Global Calibration Method (Burke et al, 2018) which operates at tract-scale, and used it to demonstrate successful processing of data from Hyper Suprime-Cam. They also demonstrated tools for tracking simulated sources through pipeline processing and assessing how well their properties can be recovered.
A monitoring and diagnostics console was developed for the Qserv database system, and was deployed at NCSA. ADQL [1] queries with common spatial constraints are now automatically optimized by the TAP [2] /Qserv service stack (a special “hinting” syntax is no longer required). Asynchronous request capability was added to the SODA [3] image-cutout service.
Preliminary installation of systems for Base Data Center database services, authentication & authorization, storage, data transfer, support of the Auxiliary Telescope System, and of the Engineering and Facility Database (EFD) were completed in Chile.
All CIENA transponders on the Spectrum path were configured and activated on August 14th. The figure below, included in the Docushare version of this report, taken from the CIENA Blue Planet management tool screen shows three paths: Boca Raton/Florida to Fortaleza/Brazil, Boca Raton to São Paulo/Brazil and Fortaleza to São Paulo. Each path was activated with a 200G channel over the MONET submarine cable system.
On August 28th, Ciena 6500 Dense Wavelength Division Multiplex (DWDM) nodes in Boca Raton and Miami were activated using the CrownCastle fiber. At the end of the activation, using CIENA’s native OTDR [4] system, the total fiber distance between MI3/Boca Raton and MI1/Miami was measured to be 106 km (66 miles). With these activations, the LSST path from Miami to Boca Raton and Boca Raton to São Paulo were completed.
[1] The International Virtual Observatory Alliance’s Astronomical Data Query Language; http://www.ivoa.net/documents/latest/ADQL.html
[2] The IVOA’s Table Access Protocol; http://www.ivoa.net/documents/latest/TAP.html
[3] The IVOA’s Server-Side Operations for Data Access protocol; http://www.ivoa.net/documents/SODA/
[4] Optical time-domain reflectometer