The DM monthly status report covering June has been posted in Docushare, Collection-4640. For your convenience I am pasting high-level summary and project status below. Direct link to the full report (pdf): http://ls.st/35l
High-level Summary
Data Management development continues to track the Fall 2016 rolling-wave cycle plan. We have completed the initial expansion of the 1000-core Verification Cluster for development and QA activities. A second implementation of simultaneous astrometry capability (“meas_mosaic”) has been added to the science pipelines code. Development of science user interface Firefly widgets for Jupyter notebooks was started; these will power QA and Level 3 visualization capabilities. The Base site Data Centre architectural plans have been completed with minor issues in the cooling and electrical designs to be finalized. The International long-haul network link is ahead of schedule, with 100Gb circuits in place.
The DM leadership continues with the replan of future activities to respond to slower than expected initial staffing ramp-up and to improve the construction plan in the datacenter area. Completion continues to be projected for the end of CY’16.
Project Status
Selected technical accomplishments:
- The Architecture Team supported preparations for the Director’s Review, including refining the team’s charter and resources and reconstituting the Technical Control Team (change control board). The Deputy System Architect also enabled substantial portions of DM code to run under Python 3 (as well as Python 2) in preparation for doing the same for the rest of the code.
- The AP Team updated all tests to use the HTM based reference catalogs, replaced jointcal StarSelector with the LSST StarSelector, and quantified the impact of assuming a static matching kernel on the decorrelation algorithm.
- The DRP Team converted major parts of the fundamental infrastructural code underlying the Science Pipelines to make it work with pybind11 in place of SWIG. The team made available meas_mosaic package in the LSST software stack, meas_mosails can now perform simultaneous astrometric and photometric calibration. Aperture corrections have been reworked to take advantage of the newly available catalog-based calculation plugin system (see May 2016 report). This is the first major use of this system within the Science Pipelines.
- The SUIT Team finished evaluating the old SUIT requirements, started re-writing draft SUIT system requirements, fixed most of the bugs reported by IRSA tests, and started experimental development with Jupyter widget to use Firefly visualization capabilities. Gave demo at SciPy conference.
- The DAX Team located the SDSS Stripe 82 DRP products from the S13 runs at both NCSA and IN2P3, assessed the products to verify full survey coverage, and began work to assemble and prepare the data for ingest at the PDAC. L1/AP database prototyping work continued, and incremental refinements were made to Qserv shared scans and the Data Butler.
- Within the Net&BS Team, the long haul networks are progressing at a good rate. The last part to put into place is the US section from Miami to NCSA which is being addressed in conversations with ESnet. The Chilean backbone is expected to be in place early 2017 and the International link is ahead of schedule with 100Gb circuits in place. The Base site Data Centre architectural plans have been completed with minor issues in the cooling and electrical designs to be finalized.
- The NCSA Team made significant contributions to the LSST Concept of Operations and Operations Use Case Report for LSST operations planning. The team produced a final draft of the concept of operations of the unified LSST authentication and authorization system with the LSST ISO for submission to the LSST CCB, as well as a first draft of the Batch Production Services Concept of Operations, for DM replanning. Evaluation of eight workflow systems has been completed. A representative attended the XSEDE 2016 Conference to gain further insight on current use of workflows in HPC environments. The team also completed initial expansion of the 1000-core “Verification Cluster” for development and QA activities; the system is in early “friendly user” mode.
- The SQuaRE Team converted the validate_drp package into a prototype metrics framework.