In writing docs for our development workflow, I realize that there are at least three workflows for developing the stack
eups distrib install a recent tagged release and then clone packages as needed and do the eups declare and setup. This is described on a community posting and on confluence.
Use lsst-build directly. I know of one person who actually does this, but I donāt think we should encourage it
Of course, these methods do share some commonality, but Iād ideally like settle on a single, clear, nearly fool-proof development workflow document that I can give to new hires and the open source astronomy community. Of course, deeper docs can always capture alternative workflows and tweaks.
Iāll confess that Iāve already started on the lsstsw method, since I like the approachability of having all the code cloned, and it doesnāt rely on a weekly tagged build (which seems unusual coming from a āpull masterā GitHub Flow world). At the same time, I realize that lsstsw was really made for the CI environment, and doesnāt have much in the way of human ergonomics.
I also donāt have the day-to-day stack development experience that the UW and Princeton crews have.
So Iāll open it to you: what method should we advocate in a developer workflow guide? Please vote in the poll. If you have a nuanced answer, or have a fourth way, submit a new post outlining it.
Do we think those have the same answers? lsstsw is for somebody who needs to be living life on master, developing multiple packages together and ensuring that the work is not broken by others and giving you confidence that what you have just pushed to your ticket branch is going to be a state reasonably close to what the CI is going to use to test it. eups distrib is always trailing edge a bit and you have to wait for new versions of the stack to be published before you can try out a feature added in the last week (if we have weeklies made available).
lsstsw has the added bonus of giving you easy access to all the git repos.
In so much as theyāre both groups coming in without pre-existing workflows and are wanting the best advice we can give.
Iāve thought that perhaps we need a ālsstsw for humansā that wraps a lot of the various eups-related chores, scons building and test running into a command line app. But I also realize itās probably a bad idea to introduce yet another layer on our toolchain without a clear endgame.
Improving our build toolchain is something to address in another post; for now we need to doc what weāve got.
I think this is a role lsst-build was supposed to fill back when @mjuric started building it, but like so much of our developer infrastructure it never really got developed past ābarely good enoughā, and since it was developed further as the backend for buildbot, lsstsw actually became easier for developers to use than lsst-build itself.
I voted for eups distrib (which puts me in a minority of one, for now at least). However, let me attempt to be more nuanced:
The eups distrib workflow description linked to above is complex, because it discusses tagging, declaring, and so on. Those might be convenient, but, in fact, are not necessary. At itās heart, this method boils down to three commands: eups distrib install, setup, scons. My contention is that all developers have to be conversant with those before they can effectively work with the stack, regardless of whether they also choose to adopt higher level tooling: eups distrib because it is our primary documented method of installing the stack, while setup and scons are required whatever workflow is adopted (@rowenās post, linked above, also describes their use).
Of course, there are downsides to a pure eups distrib workflow: Iām quite happy with ā indeed, would encourage ā the idea that we should describe the fundamentals and also describe higher-level tools that can make the practising developerās life easier, but I donāt think that means we can avoid covering the basics.
You do have to consider your target audience though; are you including qserv in this conversation? We discovered recently that the lsstsw tools donāt work for the qserv (or xrootd only?) workflow, at least not without some fancy footwork. @fritzm can provide better details on that I think.
I am not familiar with the lsstsw tool, and so my working model when working with xrootd and qserv has been distrib install + clone + eupspkg + tag + setup.
Nate came back from bootcamp with a cool incantation for directly declaring a git clone to the local eups. That seemed appealing because I could potentially skip the eupspkg parts. But of course, for xrootd and qserv the git tree is quite different from the binary install tree that eups in the end expects. So you canāt do that for those guys ā you have to use eupspkg {install,decl} as well.
Iām looking forward to peeping the videos from boot camp to learn more about the lsstsw tools and see how/if they could improve the workflow Iām currently using.
WRT the videos, I think I picked up the lsstsw workflow in the tutorial sessions (not during the recorded presentations); Iām not sure there will be a lot in there for you re. this. However, Iām sure you could talk to any number of people including folks at UW, specifically @rowen for more info.
lsstsw has the added benefit of being able to build any ticket branch or combination of ticket branches. This has important advantages:
If one is working on a ticket that spans multiple packages, one can update all dependent packages to the current master (e.g. to pull in a needed fix) and continue oneās work.
ā¢ It is easy to test ticket branches, which can be quite important on a system not yet supported by Jenkins.
I completely agree that developers need to be able to use those tools, and even though this isnāt my primary workflow, I do still use it frequently, especially on the HSC side where Iām more likely to be making a small bugfix to a released version.
The reason Iād advocate lsstsw (or ideally something like it but better) is that I think itās bad for developers to rely on released versions as a part of their default workflow, because that inevitably leads to version incompatibilities, and that leads to invalid bug reports (e.g. segfaults from ABI incompatibility) and unnecessary blocking (developer A canāt get work done until developer B has time to make a release). When you donāt build the whole stack from source yourself regularly (even if you know how), it becomes much harder and slower to do it when you need it, and you need it pretty often.
For the āwork on an issueā sort of application, what you call the āeups distrib workflowā is very convenient, especially in an environment where you share a stack ā and thatās going to be ever commoner as we work with larger data volumes that you donāt have on your laptop.
Many (and later most?) users will not need to check out a complete distribution of their own on a shared machine, but thatās what the lsstsw approach assumes.
The latest weekly is already installed for everyone to use (someone runs a cronjob to do the āeups distrib installā?). So a user who wants to work on a package just has to follow the patten which Jonathan references [and which I donāt know how to embed here]
and they are away. The package in question could be level 3 that needs a recent release, or it could be something way down in the stack ā although that is more likely to run into API/ABI problems. We should be checking for ABI changes, but thatās another matter (lost somewhere on trac).
Itās not clear that ālarger data volumesā necessarily implies shared machines or shared stacks. The NCSA Nebula cluster is going to be able to give each developer a separate machine with access to shared large data. (We can make a shared stack available as well, of course, but it is not mandatory.)
Perhaps be worth having a second workflow document describing how to develop against a shared stack (given itās current importance pre-Open Stack cluster).
I agree that it need not, but it opens up the possibility of shared stacks and I think we should take it. End users absolutely need it (āLook at this DECam rerunā shouldnāt start with, ābut first install the LSST stackā), and I believe that itās a very valuable mode for many/most developers. There will always be exceptions, but given that we have to support it for end users, letās take advantage of the possibility.
I think that large data volumes do imply that working on laptops will become less common.
Iām not voting yet (too new to have a strong opinion), but Iām about to re-install via the lsstsw method which I have a slight preference for right now.
That said, I donāt understand how two of the points made above are relevant here:
Users and developers are almost certainly going to have different methods. Most users probably shouldnāt even touch eups, and just conda or pip install lsst (or just use the stack thatās already installed on their shared system). On the other hand, developers having a full checkout of the entire stack that they develop inside of seems like a clear workflow to me.
I donāt see how larger data volumes are relevant either: the software stack is separate from the data, and I thought the whole point of the butler was to make the code completely data-location agnostic. In the worst case, afw_data (for example) would live on a network drive, etc. and be the only package that gets setup differently. What am I missing?
