On Thursday, 2021-01-14, we will update our build infrastructure to use the rubin-env conda-forge metapackage (DM-27005). This is the result of extensive work over many months by Gabriele Comoretto (on scripts and Jenkins) and Brian Van Klaveren (on conda and version conflicts). The d_2021_01_15 daily and w_2021_04 weekly will be the first tagged releases to incorporate this change.
On one hand, this change is intended to have minimal effect on users of the environment and the Science Pipelines.
This change is the equivalent of releasing a new hash version of the scipipe_conda_env package that we have been using to define the conda environment used by the Science Pipelines. In addition to upgrades of a number of packages and moving to Python 3.8 (RFC-725), this new environment will include two requested packages:
as well as eups, which has been available in conda-forge for some time.
miniconda 4.9.2 based on Python 3.8 will be used instead of 4.8.2 based on Python 3.7. This improves the ability of conda to solve environments, but the miniconda Python version is not actually directly related to the Python version used within the environment.
We will continue to use our three main methods of installing the Science Pipelines and its associated conda environment: newinstall.sh, lsstsw, and Docker containers. There will be slight changes to the arguments of these if non-default environments are desired: instead of a git hash, we will be using a semantic version number for the metapackage, currently 0.1.5. This version will appear in the environment name in place of the hash that was there previously. For backward compatibility, the lsstsw deploy script can still take a hash version of scipipe_conda_env. For newinstall.sh, previous tagged versions of the script can be used.
On the other hand, the metapackage allows us to unpin the versions of most dependencies while using the conda-forge CI infrastructure to ensure that a consistent set of those dependencies is always available. This unpinning makes it much easier for users to install new conda packages into the environment along with rubin-env. The use of a metapackage also makes it easier for other users to define their own packages or metapackages that build upon the rubin-env environment. This is expected to substantially improve the conda environment situation for the Rubin Science Platform, Sims, CC-IN2P3, and the lsst-devl “shared stack” at NCSA.
Over time, we may need to constrain versions of more dependencies; our Jenkins clean builds will give us warning that this may be necessary.
Unpinning dependencies changes the reproducibility of the environment in which the Science Pipelines software runs. We will have two levels of reproducibility:
- For most developer uses, we will use
rubin-envwith unpinned dependencies. - For production runs, we will have the ability to use a completely pinned environment (as we do today with
scipipe_conda_env) that has been vetted and output by Jenkins and saved with a tagged release.
If necessary, users can use standard conda commands to dump the exact dependency versions they are using to enable reproduction of problems.