DM python and associated packages version baseline change

Data Management DM Notifications
, , ,
#1

The Data Management Software stack relies on a Python installation containing numpy, astropy, and matplotlib. If you install a stack with newinstall.sh or lsstsw these packages will likely have been installed as part of that installation and will have versions pinned by the installation process.

We have just changed the versions that will be installed as part of that process so if you reinstall you will get Python 3.6, numpy 1.13, astropy 2.0 and matplotlib 2.0. We are also taking this opportunity to switch to setuptools to v36 to enable us to support current versions of flake8 in our infrastructure.

At the start of October we will update the version checking in our stack build such that at least those versions of Python, numpy, astropy, and matplotlib must be available to the build. The build will abort if that is not true. This will affect anyone who installs weeklies or uses lsstsw after that date. The imminent v14.0 release will use the older dependencies.

For Python 3.6 and 2.7 users you will need to type the following if you wish to continue to use your current installation with newer releases:

$ conda update astropy numpy matplotlib setuptools

For Python 3.5 uses, Python 3.6 is not ABI compatible with Python 3.5, therefore stack packages that use the Python/C++ interface will need to be rebuilt after updating. If you wish to reuse your installation despite this breakage you can update your Python versions in place with:

$ conda install python=3.6
$ conda update astropy numpy matplotlib setuptools

This will break all previous builds in that installation, so for this reason we recommend that you start a new Python3.6 stack install at some point in September. We apologize for the inconvenience caused by this ABI breakage.

If you are using eups distrib with the weeklies, you can rebuild older weeklies in your new installation. All weeklies produced since the end of June (from w_2017_26) are compatible with numpy 1.13.

When using newinstall.sh please check that you are using a version that installs Python 3.6 before attempting to install new packages. The version shipped as part of v14.0 will install Python 3.5.

1 Like
Changes to baseline software versions
Why do we use the nomkl version of the python modules?
#2

Thanks for posting this and making clear about the 3.6 ABI changes.

Just to be clear, for those of us using lsstsw, we should be able to follow the usual instructions and get a new installation that’s ready to go?

#3

Yes.

1 Like
#4

I’ve attempted to follow the instructions to get a shiny new python3.6 stack, but daf_persistence is failing all over the place because it expects python3.5. Is it necessary to start an entirely new lsstsw from scratch for this to work? I (perhaps naively) thought blowing away everything in lsstsw/builds/ would do the trick after I installed python3.6 and updated the four packages with conda.

#5

That will be the ABI problem where daf_persistence is trying to load utils or something.

Wrong directory. That directory has all the git clones in it. The ones you wanted were stack/DarwinX86/ (or the Linux equivalent) and stack/ups_db/*.

2 Likes
#6

This update to the minimum version checks has now been released (not quite October). DM developers may now use features in these versions of numpy, astropy and matplotlib that are not present in earlier releases.

1 Like