Python and Conda environment update

Data Management DM Notifications
, ,
#1

Yesterday we updated the baseline for the science pipelines conda environment. This means that the next time you run newinstall.sh or lsstsw deploy you will get the updated environment. The full set of versions defined by the new environment can be found here but the main changes are:

  • The Python baseline is now v3.7.
  • Numpy 1.16.2
  • Astropy 3.1.2
  • matplotlib 3.0.3

Furthermore the deprecated package is now available in the environment and can be used to implement the new deprecation policy.

Now that we have made this change you will need to reinstall all your local stack builds since nothing will be compatible with the previous builds. @swinbank will be creating a new shared stack.

Next week I will update the corresponding eups stub packages to force these minimum versions, at that point all builds will fail without updating.

Note that this update has led to us discovering that macOS Sierra compilers no longer build the stack. We have disabled the macOS builds from Jenkins stack-os-matrix and the Mojave build node will be used for binary releases. We will make an announcement when the Sierra nodes have been updated to Mojave and the Jenkins jobs restarted.

#2

Now that we have made this change you will need to reinstall all your local stack builds since nothing will be compatible with the previous builds.

Checking on this statement: is bin/deploy && rebuild lsst_distrib sufficient, or do I need to install an entirely new lsstsw first?

#3

You need to ensure that everything that uses python rebuilds. In theory you can redeploy over the top and get the new env but that would lead to all previous builds in that stack not being functional anyhow (and you need to force it to rebuild something like python). It takes about the same time to start from scratch.

#4

Except for all of the pip installs I have, and the astropy-dev conda env. Deleting the old builds isn’t an issue.

#5

You can definitely try to update the packages in place, it’s just not something that is automated. Then eups remove the python stub and things should work.

#6

PS Updating the python and numpy versions are the super important parts of it.

#7

Was the conda version also updated to 4.6?

#8

No, but it’s trivial for you to do that. The problem is that there is no miniconda 4.6 installer. When you run the install it does give you instructions immediately on how to do the update and updating conda has no bearing on package builds.

#9

Ah, OK thanks.

#10

For those who like me who want to try to upgrade in place, these are the steps I did, from within my lsstsw installation directory (after making a copy of it so I could roll back if necessary):

# remove old files and their eups tables: they will be incompatible with the updated conda env
rm -fr stack/Linux64/*
rm -fr stack/ups_db/*
# update the bin/deploy script and conda env info
git pull
bin/deploy
rebuild lsst_distrib

I found if I tried to do the conda update first, I got an error when I ran bin/deploy about there being nothing to install and a suggestion to run conda clean --help. Doing the above commands appears to have worked: I now have an updated stack with the new baseline packages, but still have my pip packages installed (whether they all work is a separate question).

Conda environment refresh
#11

If you are on Mojave and having trouble building validate_drp because of a segv in the json load test, the problem is that there seems to be an incompatibility between the tkinter plug in and Mojave. This is triggered by the default matplotlib backend being TkAgg. To “fix” this problem you may need to change the default backend to Agg by adding (or editing) the file ~/.matplotlib/matplotlibrc and adding a line:

backend : Agg