This page describes a method of installing and maintaining a DM software stack that allows you to easily keep up to date with the current master of packages and to push your own changes. It was updated 2015-11-04 to support git-lfs packages such as afwdata.
- Set up a git credential manager, e.g. as per afwdata’s README.md.
- (alternately: edit the file lsstsw/etc/repos.yaml to change all instances of https://github.com/lsst, except afwdata,to firstname.lastname@example.org:lsst. This works fine if you don’t plan to push to afwdata, and doesn’t require using the credential manager.)
- Clone the lsstsw package into the location of your choice:
git clone email@example.com:lsst/lsstsw.git
- Install lsstsw and rebuild lsst_apps using these instructions. Note that this pulls over all of afwdata, which is a lot of data. If you already have a cloned copy you may be able to avoid this by making a symlink to it from
- Eups tag the current build as “current” using command:
eups tags --clone bNNN
bNNN is the eups tag that lsstsw used for the packages installed by the
To work on one or more packages, start by making a fresh build (if you have not gotten one recently) using rebuild:
If this complains that it cannot clone some packages then you will have to update repos.yaml; the simplest way to do this to specify the
-u flag in the rebuild command.
eups tags --clone bNNN current as described above
Next declare the git clones of the packages you will be working on to eups. The git clones are in
$LSSTSW/build and I suggest you declare them with version
git and a eups tag of your username (since that tag is permitted by eups without altering the eups configuration). For example, user
rowen could issue the command
eups declare -r $LSSTSW/build/afw afw git -t rowen.
Now edit your code and rebuild manually, using
scons as needed. Having eups-tagged your packages, you can set them up using:
setup -r . -t your_username.
Warning: the rebuild command will erase any changes you have not pushed. Do not run the
rebuild command while developing and testing your code unless you really need it. Before you run
rebuild carefully verify that you have pushed (not just committed) all your changes.
When you are done with your work:
- Clean your commit history using
git rebase -i $HEAD~N where N is the number of commits.
- Get your changes reviewed.
- Clean up your code as required by the review and clean up the commit history again.
- Merge your changes to master and push all changes.
- Remove your username tag from the packages. For example to remove your username tag from afw:
eups undeclare afw git -t your_username.
- Build a fresh copy of the stack using
rebuild lsst_apps followed by
eups tags --clone bNNN
- Be careful not to overwrite your work with a “rebuild” command. Always commit and push before rebuilding, or do your work on clones that are not in
- Be sure to clone tags if you want your new build to be current, because otherwise the default “setup” command won’t act as you expect.
- Be careful when trying to run packages in the build directory (rather than installed versions) because “rebuild” only builds what it has to, so some of those directories may not have been built.
- You should usually specify “-u” with the rebuild command to update your repos.yaml file. Otherwise you risk the build failing with a complaint about missing packages.
How does this ‘lsstsw’ method of curating a development stack differ in practice from using
lsst-build (described in https://confluence.lsstcorp.org/display/LDMDG/The+LSST+Software+Build+Tool)? Should a stack developer always use the lsstsw workflow your describe, or are there cases where
lsst-build should be used directly?
Well, I use
lsst-build directly but maybe I’m just special. I think it was because I couldn’t find the right documentation when I was trying to find out what to do so I simply looked around the
lsstsw checkout and the documentation in the code that I could most easily follow was in the
I should add that it took me a few times before I realised that this scheme trashes your entire working tree and you have to push to the remote repo. I was somewhat confused the first few times when I had done work on a ticket branch and asked
lsstsw to build it, only to find that my branch had disappeared (luckily I knew the SHA1 so I could find it again).
I use a much less magic approach (magic is of course in the eye of the beholder). When I want to get some work done, I build a recent weekly (I’ll use a binary when I can) :
eups distrib install lsst_apps -t w_2015_36
Then if I want to work on e.g. afw,
(where I have a git clone)
setup -r . -j
scons -Q opt=3 -j 6
and I’m good to go. If I need to work on more than a product or two, I tag them:
eups list -t rhl
eups undeclare -t rhl NAME any that I don’t still need)
eups declare -r . -t rhl
setup -t rhl -t w_lsst_36 lsst_apps
I should be able to make w_lsst_36 current with a
eups tag --clone=w_lsst_36 current command, but there seem to be problems in the table file that the versions are incorrect – it’s on @mjuric’s list. If I could, then either
setup -t rhl -T w_lsst_36 lsst_apps
or a simple
setup -t rhl lsst_apps
Is this line correct? or is this the correct command?
setup -r . -t your_username
(leaves out the
setup). - when I leave off eups the command works, otherwise it says “eups setup: Unrecognized command: setup”
Just testing you . Fixed, and thanks.
thanks! I’m glad to know I’m at least somewhat sane
The work flow @RHL advocates also works, but I feel is less suitable for active daily development because
lsstbuild) provides direct support for the following:
- Pull in a needed fix, either from master or one or more ticket branches
- Rebuild a set of interrelated packages after making a fix to one of them
eups distrib install is something developers should know how to use, but I don’t recommend it for daily development because it is too limited and offers too great a chance of incompatibility between the master you are working on and the last weekly release. See also What development workflow should be advocated? [w/ Poll] for an extensive discussion.
Does your proposal work with a shared stack?
I doubt using
lsstsw to build shared stacks for development is a good idea, though it can be done. The biggest issue is that each developer would have to keep packages being worked on out of the main stack area, so that if one developer runs
rebuild it does not overwrite the work of other developers. That loses one of the main advantages to
lsstsw: being able to build all related packages on one or more tickets. Still, with careful coordination it can work.
For a shared stack that is intended for users instead of developers
lsstsw seems even less appropriate, since users should use official releases of the software, if possible.
That was what I thought. I believe that we need a workflow that works with a mostly shared stack. This doesn’t mean that this way of working will be good for all developers/situations, but if it isn’t I think we need to document and support two options.
It’s a bit tricky to do it all. Developers often require cutting edge software, and users are usually better served by official (or semi-official) releases. It is hard for me to imagine that developers and users would usefully share a stack.
I can imagine developers being able to share lsstsw stacks if we tweak lsstsw to install in a shared location but offer each developer a personal “build” directory. I have no idea if that would provide enough benefit to be worth doing. It would save some space, but it would also get very cluttered and that clutter would be hard to clean out. And a cluttered stack is hard to manage; eups slows down and even if that were fixed, the output of
list commands becomes large enough to be a problem.
Why? A specific version (or tagged version) is well defined in a shared stack whether or not some developer has other versions installed.
Russell just requested that I post this here for his memory.
When I need to clean a bunch of old builds from my lsstsw, I do the following:
rm -rf DarwinX86/*
rm -rf ups_db/*
rebuild lsst_apps, and replace the
afwdata/BLAH directory with a symlink to my
~/lsst/afwdata checkout (which is kept up to date because it’s symlinked to
This lets you clean up an overly-large eups list and free a bunch of disk space, without having to reinstall everything or damaging your miniconda install/environments.
One minor addendum: also do the same for
testdata_cfht. None of the other testdata packages is large enough to be a concern at this time.