FY17 Decommissioning Activities

Data Management DM Notifications
#1

During the month of Sept, I am focusing on the planning for procurement activities for the following fiscal year. Those details are unfolding here: https://confluence.lsstcorp.org/display/DM/DM+FY17+Procurement+Planning.

What is probably of particular interest to this audience are the services that are being decommissioned pending any input from the project. The necessity of these services is not clear; they appear to be legacy cruft. NCSA is capable and willing to extend operations of any of these services as necessary given that the need is identified. The most up-to-date version of the list is on the page mentioned above but an initial list is provided below for convenience. Please review and notify me of any concerns.

Note that any data sets distributed by discontinued services below may be available via the new /datasets file system. That plan is forthcoming.

Decommissioning Activities
The following hardware is being replaced

  • lsst10/lsstdb
  • lsst-dbdev[1-6]

The following hardware is being retired

  • lsst-suidev[1-2]
  • lsst-stor14[1-4], lsst[13,20] (NFS systems)

The following services are being retired

#2

I believe the Software Distribution Service at http://sw.lsstcorp.org/eupspkg may still be in use. Someone from SQuaRE should confirm or deny this; itā€™s possible I need to update to a new URL.

Iā€™m also not yet aware of a replacement for the Doxygen builds at http://lsst-web.ncsa.illinois.edu/doxygen, though I know one is planned and I may have missed an interim update to a new URL here as well.

#3

Confirming that http://sw.lsstcorp.org/eupspkg is definitely still in active use.

#4

@jalt whatā€™s your timeline on the retirement of http://lsst-web.ncsa.illinois.edu/doxygen ?

The replacement for that service will be https://pipelines.lsst.io; Iā€™m just working on the integration of Sphinx into the build process right now.

If that doesnā€™t happen in time, I could work with @josh on modifying our build pipeline to upload the current doxygen builds to LSST the Docs rather than copying files onto lsst-web.ncsa.illinois.edu as we currently do.

#5

These absolutely need to be retained:

These would be desirable to keep:

You need to notify the Sims group (not DM) for these:

You need to notify the Calibration and/or Systems Engineering groups (not DM) for this one:

  • Calypso Data Access
2 Likes
#6

http://sw.lsstcorp.org/eupspkg is a critical service. There are rough plans to migrate this to a new canonical URL/backend but this activity is completely unscheduled at this point.

#7

It would be unfortunate to lose the webpages served off of user home directories. I use that frequently both for distributing large files and for creating different visualization pages.

1 Like
#8

I agree with both @ktl and @ctslater that these would both be very nice to keep.

@connolly and @danielsf may have more to say about the sim services.

#9

Replaced as in the current database on lsst-db will still be available on a machine that is still called lsst-db? I recently moved all lsst10 usage in the test suite to lsst-db.

#10

Sim Data needs to be kept. That is where we host the data (SED libraries, dust maps) that get installed with the simulations software stack.

#11

If by ā€œCatSim Documentation pagesā€ you mean these confluence pages

https://confluence.lsstcorp.org/display/SIM/Catalog+Simulations+Documentation

I would appreciate keeping them for one more year, until we can replace them with LSST-the-Docs pages containing the same information.

Thanks.

#12

This discussion is not about confluence. This is entirely an NCSA activity.

#13

Please note that we should not agree to retire a service at NCSA with the expectation that it will be re-hosted at the same (or worse) support level at another institution. If the service provides a function, leaving it at NCSA should be the default.

#14

Exactly. There will be new hardware that will be ā€˜swappedā€™ in during a maintenance period. Actually it will be far more interesting than a swap because we are changing spindles.

So this is an excellent time to speak up if there are new use cases for that node. Currently I am just planning for more cores, ram, spindles.

#15

Iā€™m referring to: http://lsst-web.ncsa.illinois.edu/catsim which, I believe, points down into /nfs/lsst/public_html/catsim.

#16

Done. What about the other two links:
http://sw.lsstcorp.org/pkgs
http://sw.lsstcorp.org/pkgs/std/w12

#17

Hi Jason,

Thanks for tackling this ā€“ I wouldnā€™t be surprised to find things running that were set up in the past decade :slight_smile:

Below are my annotations. I also took a stab at an initial definition of which project role should be the contact point for questions on retirement, changes, or continued maintenance of each service. Hope this helps you with change control going forward.

Current change control process: once you cull the above list based on all comments, issue RFCs for retirement of individual services. The contact points above should sign-off on changes. Full TCT may need to weigh in on some (I tried to annotate those I thought fall into that category). In particular, ***donā€™t decommission a service w/o an RFC (our current equivalent of a CAB procedure)***. Omnibus RFCs are fine for dealing with multiple services at once.

Again, thanks for tackling this! :+1:

Hope this helps!

#18

I was thinking it was PS1 images for testing image subtraction. The decision about what to do with it is above my pay grade, but I would suggest that we have better data to test with.

#19

Panstarrs links: http://lsst-web.ncsa.illinois.edu/panstarrs (/lsst6/panstarrs)

Iā€™m compiling an updated list. See the confluence link above for the latest version. Still lots of work to do on this list. Iā€™ll post an update soon.

#20

Okay. Those are all quite ancient and bound to be out of date. They can go away.