Notes from NCSA OpenStack/Nebula users meeting

Data Management
#1

I attended the NCSA Nebula/OpenStack users meeting on 7/28/2016. They have these meetings every couple of months. If you are on NCSA’s nebula-announce mailing list (this should include all NCSA Nebula users), then you should receive an invite to these meetings.

Meeting Agenda

  1. Innovative Technology Services (ITS) discussion
  • Technology
  • Changes to Nebula
  • Roadmap
  • Any issues/other topics for discussion
  1. User Presentations
  • National Data Service (NDS) - 20 minutes
  • Innovative Software and Data Analysis Division (ISDA), Rob Kooper - 20 minutes

Meeting Notes

Usage Statistics

  • Averaging 300 active VMs at any given time
  • 11 billion gigabyte-hours (?), ~1.3 PB of data allocated
  • 33 active-ish projects, 200 users
  • Been online 12 almost months

System

Some details about the OpenStack implementation at NCSA

  • Gluster for backend system (volume)
  • Gluster is 15 storage nodes, creates shared space for instances and images
  • 5 groups of 3 (~raid 5), can lose one node per group
  • 58TB/node, closer to 68 with compression
  • Nova storage being moved into Gluster
  • “Nebula tax” -> The storage overhead needed to support live migrations
  • Implementing live migrations,
  • They’ve been thoroughly testing it (i.e. moved the same VM 1000s of times)
  • No real issues seen with this.
  • For LSST, testing Swift
  • Will run for 45 days, getting feedback from us (Steve?)

Implemented Quotas in NCSA OpenStack

These are good to be aware of in case you run into issues with a VM.

  1. Port limitations, at 50 can open up 200 ports per instance
  2. Other invisible (to user) quotas: metadata items, vcpus, injected files, bytes in injected files, total size of volumes and snapshots, security groups and rules, networks, ports you can map, routers you can create, and subnets you can create, a few more. As Swift comes online, new limits will be made (aka numbers of objects you can have)

Other notes

  • NDS (National Data Service) Uses Kubernetes+Docker+Ansible to setup their platform
  • “Hardest part of kubernetes is the setup”
  • Jenkins instances are most targeted for attack at NCSA
1 Like