This page describes a method of installing and maintaining a DM software stack that allows you to easily keep up to date with the current master of packages and to push your own changes. It was updated 2015-11-04 to support git-lfs packages such as afwdata.
Initial Installation
Set up a git credential manager, e.g. as per afwdata’s README.md.
(alternately: edit the file lsstsw/etc/repos.yaml to change all instances of https://github.com/lsst, except afwdata,to git@github.com:lsst. This works fine if you don’t plan to push to afwdata, and doesn’t require using the credential manager.)
Clone the lsstsw package into the location of your choice: git clone git@github.com:lsst/lsstsw.git
Install lsstsw and rebuild lsst_apps using these instructions. Note that this pulls over all of afwdata, which is a lot of data. If you already have a cloned copy you may be able to avoid this by making a symlink to it from lsstsw/build/afwdata.
Eups tag the current build as “current” using command: eups tags --clone bNNNcurrent
where bNNN is the eups tag that lsstsw used for the packages installed by the rebuild command.
Development Cycle
To work on one or more packages, start by making a fresh build (if you have not gotten one recently) using rebuild:
rebuild lsst_apps
If this complains that it cannot clone some packages then you will have to update repos.yaml; the simplest way to do this to specify the -u flag in the rebuild command.
eups tags --clone bNNN current as described above
Next declare the git clones of the packages you will be working on to eups. The git clones are in $LSSTSW/build and I suggest you declare them with version git and a eups tag of your username (since that tag is permitted by eups without altering the eups configuration). For example, user rowen could issue the command eups declare -r $LSSTSW/build/afw afw git -t rowen.
Now edit your code and rebuild manually, using scons as needed. Having eups-tagged your packages, you can set them up using: setup -r . -tyour_username.
Warning: the rebuild command will erase any changes you have not pushed. Do not run the rebuild command while developing and testing your code unless you really need it. Before you run rebuild carefully verify that you have pushed (not just committed) all your changes.
When you are done with your work:
Clean your commit history using git rebase -i $HEAD~N where N is the number of commits.
Get your changes reviewed.
Clean up your code as required by the review and clean up the commit history again.
Merge your changes to master and push all changes.
Remove your username tag from the packages. For example to remove your username tag from afw: eups undeclare afw git -tyour_username.
Build a fresh copy of the stack using rebuild lsst_apps followed by eups tags --clone bNNNcurrent.
Caveats
Be careful not to overwrite your work with a “rebuild” command. Always commit and push before rebuilding, or do your work on clones that are not in lsstsw/build.
Be sure to clone tags if you want your new build to be current, because otherwise the default “setup” command won’t act as you expect.
Be careful when trying to run packages in the build directory (rather than installed versions) because “rebuild” only builds what it has to, so some of those directories may not have been built.
You should usually specify “-u” with the rebuild command to update your repos.yaml file. Otherwise you risk the build failing with a complaint about missing packages.
How does this ‘lsstsw’ method of curating a development stack differ in practice from using lsst-build (described in https://confluence.lsstcorp.org/display/LDMDG/The+LSST+Software+Build+Tool)? Should a stack developer always use the lsstsw workflow your describe, or are there cases where lsst-build should be used directly?
Well, I use lsst-build directly but maybe I’m just special. I think it was because I couldn’t find the right documentation when I was trying to find out what to do so I simply looked around the lsstsw checkout and the documentation in the code that I could most easily follow was in the lsst-build command…
I should add that it took me a few times before I realised that this scheme trashes your entire working tree and you have to push to the remote repo. I was somewhat confused the first few times when I had done work on a ticket branch and asked lsstsw to build it, only to find that my branch had disappeared (luckily I knew the SHA1 so I could find it again).
I use a much less magic approach (magic is of course in the eye of the beholder). When I want to get some work done, I build a recent weekly (I’ll use a binary when I can) :
eups distrib install lsst_apps -t w_2015_36
Then if I want to work on e.g. afw,
cd LSST/afw
(where I have a git clone)
git pull
setup -r . -j
scons -Q opt=3 -j 6
and I’m good to go. If I need to work on more than a product or two, I tag them:
eups list -t rhl
(and then eups undeclare -t rhl NAME any that I don’t still need)
eups declare -r . -t rhl
followed by
setup -t rhl -t w_lsst_36 lsst_apps
I should be able to make w_lsst_36 current with a eups tag --clone=w_lsst_36 current command, but there seem to be problems in the table file that the versions are incorrect – it’s on @mjuric’s list. If I could, then either
Is this line correct? or is this the correct command? setup -r . -t your_username
(leaves out the eups before setup). - when I leave off eups the command works, otherwise it says “eups setup: Unrecognized command: setup”
The work flow @RHL advocates also works, but I feel is less suitable for active daily development because lsstsw (or lsstbuild) provides direct support for the following:
Pull in a needed fix, either from master or one or more ticket branches
Rebuild a set of interrelated packages after making a fix to one of them
Using eups distrib install is something developers should know how to use, but I don’t recommend it for daily development because it is too limited and offers too great a chance of incompatibility between the master you are working on and the last weekly release. See also What development workflow should be advocated? [w/ Poll] for an extensive discussion.
I doubt using lsstsw to build shared stacks for development is a good idea, though it can be done. The biggest issue is that each developer would have to keep packages being worked on out of the main stack area, so that if one developer runs rebuild it does not overwrite the work of other developers. That loses one of the main advantages to lsstsw: being able to build all related packages on one or more tickets. Still, with careful coordination it can work.
For a shared stack that is intended for users instead of developers lsstsw seems even less appropriate, since users should use official releases of the software, if possible.
That was what I thought. I believe that we need a workflow that works with a mostly shared stack. This doesn’t mean that this way of working will be good for all developers/situations, but if it isn’t I think we need to document and support two options.
It’s a bit tricky to do it all. Developers often require cutting edge software, and users are usually better served by official (or semi-official) releases. It is hard for me to imagine that developers and users would usefully share a stack.
I can imagine developers being able to share lsstsw stacks if we tweak lsstsw to install in a shared location but offer each developer a personal “build” directory. I have no idea if that would provide enough benefit to be worth doing. It would save some space, but it would also get very cluttered and that clutter would be hard to clean out. And a cluttered stack is hard to manage; eups slows down and even if that were fixed, the output of list commands becomes large enough to be a problem.
Russell just requested that I post this here for his memory.
When I need to clean a bunch of old builds from my lsstsw, I do the following:
cd ~/lsst/lsstsw/stack/
rm -rf DarwinX86/*
rm -rf ups_db/*
Then I rebuild lsst_apps, and replace the afwdata/BLAH directory with a symlink to my ~/lsst/afwdata checkout (which is kept up to date because it’s symlinked to ~/lsst/lsstsw/stack/afwdata).
This lets you clean up an overly-large eups list and free a bunch of disk space, without having to reinstall everything or damaging your miniconda install/environments.
. Fixed, and thanks.