Deploying rubin-env 9.0.0

Data Management DM Notifications
#1

On Friday, 2024-08-02, we updated our build infrastructure to use version 9.0.0 of the rubin-env conda-forge metapackage. The d_2024_08_03 daily and w_2024_32 weekly will be the first tagged releases to incorporate this change.

This environment marks the change of the Science Pipelines (“stack”) base container and the Rubin Science Platform container from CentOS 7 to AlmaLinux 9. This long-awaited modernization provides an up-to-date Linux foundation. Binary packages for Linux x86 should continue to be usable on CentOS 7 and RHEL 8 equivalents, in addition to RHEL 9 equivalents like AlmaLinux 9.

conda-forge has updated the MACOSX_DEPLOYMENT_TARGET to 10.13 for x86 Macs. The target remains 11.0 for ARM Macs, which are still supported via source builds for now, although we expect ARM binary packages to be available within the lifetime of rubin-env 9.

Other major changes include the additions of batoid, batoid-rubin, and danish (RFC-1010); sorcha, rebound, and assist (RFC-1011); colour-science and openCV (RFC-994); python-confluent-kafka 1.9 (but not 2+, due to issues with Telescope & Site Software) (RFC-964); jax (RFC-1028); pkg-config (RFC-1034); rucio-clients (DM-39804), and s3fs (needed for infrastructure). clang-format was added to rubin-env-developer (RFC-974).

Dependency version updates include cfitsio = 4.4.0, flake8 = 7.1.0, log4cxx = 1.2.0, numpy = 1.26.4, parsl = 2024.4.29, ruff = 0.5.2, starlink_ast = 9.2.11, eups >= 2.2.8, moto >= 5, matplotlib-base >= 3.9, pybind11 >= 2.12, pytest >= 8.2, and scipy >= 1.13 (but <1.14). Minimum versions have been put in place for all packages (RFC-989), guaranteeing functionality for developers.

The non-working 0.17.0 version of sphinx-automodapi was excluded from rubin-env-developer.

Binary packages that built under rubin-env 8.0.0 will not function under rubin-env 9.0.0, as indicated by the semantic versioning. Sources will often continue to work. Similarly, because of the library changes, binaries built under rubin-env 9.0.0 are not guaranteed to function under previous versions, and sources for 9.0.0 may not build under previous versions.

Thanks to the entire Build Engineering Team (Tim Jenness, Ross Ceballo, Matthias Wittgen, Arianna Ranabhat) and Eli Rykoff, with occasional help from Matt Becker, for their contributions to getting this out the door.

3 Likes
#2

Yay! Nice to see this is out. Would lsstsw rebuild upgrade local installations from 8.0.0 to 9.0.0?

#3

No, the lsstsw rebuild command is for building from sources. In the lsstsw ecosystem, you need to use bin/deploy to create a new environment and then source bin/envconfig in a fresh shell, possibly with an option to select the proper environment, before doing a rebuild.

1 Like
#4

I think you previously mentioned that, along with the Linux upgrade, the Science Pipelines containers would switch from newinstall to lsstinstall. Is this still the case, or has it been dropped?

#5

The installation of rubin-env in the container is indeed currently done with lsstinstall rather than newinstall.sh as was done previously. This should be transparent to most users, as the loadLSST.sh scripts will set environment variables accordingly, but anyone who was looking for specific paths in the container may find that they have moved.

#6

Per RFC-989, we should freeze (tag?) versions of meta.yaml and conda_build_config.yaml for reference.

#7

Yes, I forgot to do that.
The canonical versions of the fixed and minimum dependencies in rubin-env 9.0.0 that developers can rely on are:

1 Like