Shared stack on LSST development machines

Data Management
,
#1

Per RFC-156 and DM-5435, we will shortly be providing a new shared stack installation on our common development machines (primarily lsst-dev). The aim is to provide a convenient and maintained version of the stack which new and experienced developers alike can use with minimal setup overhead.

To this end, the stack will contain the weeklies (ie, tags following the pattern w_20XX_YY) of lsst_distrib. It will be based on a Miniconda Python interpreter. It will provide the standard lsstsw-required Anaconda packages (list here) and, in addition, will include Jupyter.

Questions:

  • Are there top level products other than lsst_distrib which should be included? (Something for sims, maybe?)
  • Are there other Miniconda packages which should be included?
2 Likes
Using the stack on lsst-dev
#2

It’s just going to be lsst_distrib? Not lsst_apps?

#3

lsst_distrib includes lsst_apps, as well as ctrl_execute, ctrl_platform_gordon, ctrl_platform_lsst, datarel and obs_subaru. I can easily add other top level products if that would be helpful.

#4

OK! For some reason I was under the impression that lsst_apps wasn’t included in lsst_distrib.

#5

Including lsst_sims would be great if it’s not much additional overhead.

#6

I do think we have enough obs_decam users to include it here.

#7

obs_decam

I’ve no objection to including it, but it doesn’t seem to be given a weekly tag.

@frossie – any chance we could add lsst_ci to the weeklies?

#8

Please be sure to include pipe_drivers. Don’t think that’s in lsst_distrib yet.

#9

I don’t think we want to add this independently – and I’m quite sure it’s not currently getting a weekly tag. As and when it gets added to lsst_distrib we’ll pick it up automatically.

1 Like
#10

John, this will be super useful!

Could you use the full Anaconda instead? It would solve the many of the “which extra packages” questions.

#11

lsst_ci

which requires

obs_subaru
obs_cfht
obs_decam
obs_lsstSim
obs_sdss

testdata_subaru
testdata_cfht
testdata_decam

And which, as you requested to @frossie, should also get a weekly tag.

(I gather you’ve already added this implicitly, but I wanted to explicity state the request.)

#12

The shared stack will only include products which are being tagged (preferably as weeklies, but any other tag that we’re publishing via eups distrib would work). I’ve opened DM-5661 to request a weekly for lsst_ci.

#13

It’s not a drop in replacement, since I’m currently using eups distrib to install Miniconda and I don’t think we provide an equivalent for Anaconda. I’ll spend a few minutes seeing how easy it would be to do something else instead.

#14

We are thinking of replicating this at CC-IN2P3. Are you planning to provide some tools to do this automatically ?

#15

Yes, absolutely. There’s already work-in-progress code here; it’s sadly lacking in documentation at the moment, but when it’s done, I’ll write a short note describing how it all fits together.

#16

I think that conda install --yes anaconda will turn miniconda into it’s more fearsome sibling.

#17

The promised stack is now available on lsst-dev. Initialize it by running:

$ source /ssd/lsstsw/stack/loadLSST.bash
$ setup lsst_apps

This should provide quick & convenient access to all 2016-vintage weeklies of lsst_distrib, as well as Anaconda Python with a comprehensive set of packages. Other LSST products (obs_*, pipe_drivers, etc) can be added as and when they start getting tagged for weekly builds (DM-5661).

Updates to the user-facing documentation on setting up a development environment on lsst-dev is available from branch tickets/DM-5435 of dm_dev_guide.

The stack is automatically updated using the shared_stack.py script in branch tickets/DM-5435 of shared_stack. This code is written for internal convenience and is not advertised or supported as general purpose means of installing the stack, but may be of use to some.

Feedback welcome.

5 Likes