DM Highlights Weeks of 2015-11-18 to 2015-12-01


  • Held two-day meeting of the Joint Coordination Committee with CC-IN2P3, including a special HPC facility coordination meeting involving representatives from Argonne, NERSC, and Blue Waters.
    • Intel Xeon-Phi-like architectures are looking more probable for big systems.
    • NERSC Shifter provides containerization with increased security with docker compatibility.
  • DOE is investing in OpenMP.
  • Progress is occurring in the area of object stores; Argonne is interested in helping evaluate these.
  • Back-ported HSC-developed parallelization code, including job submission and inter-process MPI messaging, as LSST ctrl_pool package. This is expected to be a stopgap until process execution middleware with layering of functionality is available.
  • Changed the defaults for creating coaddTempExps (single-visit images warped and trimmed to a coadd patch) to subtract the background and use a lanczos3 kernel.
  • Improved overscan correction for DECam data to handle “bias jumps”. Other ISR improvements are ongoing; “glowing edges” remain to be fixed.
  • Published technical note about technical note publishing platform using that platform.
  • Held differential chromatic refraction literature overview and discussion of ways forward.

Other items:

  • Decommissioned old servers.
  • Upgraded ActiveMQ message broker to 5.12.1.
  • Fixed HealPix-based skymaps.
  • Increased quotas on Nebula OpenStack cluster to accommodate increased usage.
  • Updated DM risks in risk register.

Continuing work:

  • Progress on detailed design of La Serena data center.
  • Progress on detailed design of Level 1 systems, including investigation of possibility of elimination of the network outage buffer.
  • Progress on operations proposal including draft WBS for operations.
1 Like

@jbosch, @price Quick question – where can I learn more about ctrl_pool and how to use it to run LSST pipelines?

I’m afraid it can’t be used directly; it’s just a framework that allows you to define new higher-level tasks that run several other tasks. We haven’t yet transferred over any of the HSC-side high-level tasks, but I think that’s next on @natelust’s to-do list. I think writing documentation on how to create new ones is part of that work.

OK, so this is primarily (relatively low-level?) HSC support. It may be good to mark it as such in a README.

Thanks (& do feel free to ignore me over the Holidays :slight_smile: )!