We’re announcing a preview of our stack binary releases for OSX and Linux (centos 6 & 7) platforms.
Try it out
Users may try out a binary installation with the
w_2017_35 weekly release.
First, ensure the platform pre-reqs are installed.
Then get and run the revised newinstall.sh script:
curl -sSL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh | bash -s -- -cbt
-t flag enables “tarball” installations.
Then load the environment and install the latest weekly as usual:
. ./loadLSST.bash eups distrib install lsst_distrib -t w_2017_25 -vvv # required to fix python shebang lines curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python # optional -- run the stack demo curl -sSL https://raw.githubusercontent.com/lsst-sqre/buildbot-scripts/master/runManifestDemo.sh | bash -s -- --small --tag w_2017_25
eups distrib install preferentially installs binaries, and falls back to source installations of packages.
What is a “tarball” binary package?
Many DM developers are already familiar with the EUPS distrib source, or
eupspkg, packages. An EUPS distrib binary, or
tarball, package is very similar in functionality to
eupspkg, except that it contains the verbatim contents of an installed EUPS product rather than source code which requires an extra build step.
EUPS distrib has support for searching through multiple
EUPS_PKGROOTs. It’s possible to transparently mix the usage of
eupspkg from different repos.
What binary builds are available?
In general, tarballs for the
lsst_distrib “meta” product and all of its dependencies are created as part of the automated DM weekly tag/release process. EUPS distrib does appear to have the ability to list which tags are available in a repo (
eupspkg). It is possible to browse an HTTP accessible repo to discover which tags are present. Eg., https://eups.lsst.codes/stack/redhat/el7/gcc-system/miniconda3-4.2.12-7c8e67/ where the available tags are files that end with
Binary object code is inherently platform specific, which means a separate binary build has to be produced for at least each operating system. In some cases a single binary will work [as intended] on multiple releases of an operating system, depending on lib/ABI changes. This is currently the case for our macOS binaries, which are known to work on 10.11 (“El Capitan” and 10.12 (“Sierra”), but not for the Centos/EL binaries. The major.minor release of python linked against also requires a separate set of binaries. Similarly, the the compiler used may also require separate binary releases due to issues with combining object code generated from different releases that could prevent an end-user from building against a binary distribution.
operating system + compiler / python major.minor version matrix
At present, binaries are being built with a single compiler per operating system version. However, we anticipate that compiler versions may change in the future and that the end-user should be protected from installing EUPS binary products built with different compiler versions. Thus, the compiler is treated as a separate axis in the matrix of binary builds.
| | el6+devtoolset-3 | el7+gcc-system | osx+clang-800.0.42.1 | |------------|------------------|----------------|----------------------| | py2.7% | X | X | ^ | | py3.5% | | X | ^ |
- % In reality, these are specific miniconda distribution versions
- ^ not currently being produced – coming soon
Actually, no. Since various DM software packages link against libraries that are install from conda packages, the specific of the python/conda package env the build was run is needs to be tracked as well. This is currently being represented as a 6 character abbreviated sha1 has of the
lsst/lsstsw repo that presents the specific conda environment files used to bootstrap the build environment.
My head just exploded…
newinstall.sh will attempt to figure out all the parameters needed to use binary tarballs when you are on a supported platform. The hope is that most binary consumers will use
newinstall.sh and not need to worry about the details. In fact,
newinstall.sh is the only supported vehicle for consumption of the binary tarballs at this time.
A convention for a hierarchy of EUPS distrib repos has been worked out with the hopes of enable a
EUPS_PKGROOT to be constructed heuristically.
All repos need to live under a base URL. At the present time, the base URL for published EUPS distrib product is https://eups.lsst.codes/stack . A source or
eupspkg repo may live at
<base url>/src. At present, that would be https://eups.lsst.codes/stack/src .
There are [currently] ~4 parameters required to construct the path under the base URL for a binary release:
| parameter | example | |-----------------------------------------|---------------------------| | `<os family>/<platform>` |`redhat/el7` | | `<compiler>` | `system-gcc` | | `<miniconda py>-<minconda version>` | `miniconda3-4.2.12` | | `<lsstsw abbrev-ref>` | [7c8e67](https://github.com/lsst/lsstsw/tree/7c8e670ce392ea11c64b4c326a130d6fa7f2d489/etc) |
Putting those values together would construct an
EUPS_PKGROOT URL of https://eups.lsst.codes/stack/redhat/el7/gcc-system/miniconda3-4.2.12-7c8e67/
We can also have EUPS transparently fall back to using
eupspkg if a
tarball build of a specific product/version isn’t available. Eg.,
How to list
$ eups distrib path https://eups.lsst.codes/stack/redhat/el7/gcc-system/miniconda2-4.2.12-7c8e67 https://eups.lsst.codes/stack/src
Can’t you make this less complicated?
We tried hard to hide more of the complexity but the nature of DM’s custom build tool-chain proved to have fatal edge cases when packaging up a complete miniconda/python environment as an EUPS distrib product.