Using Python's venv with newinstall's Anaconda

Data Management
#1

This post describes how to layer a Python virtual environment on top of the Anaconda Python that comes with standard LSST Science Pipelines installations like newinstall.sh and lsstsw.

Background

The motivation for this is that you might want to install or develop a regular (as in, PyPI) Python package to use with the LSST Science Pipelines, but you don’t want to install that package and its dependencies in the miniconda site-packages directory. (If you do pip install a Python package into the LSST Science Pipelines’ miniconda site-packages, it can be hard to completely uninstall it and its dependencies later).

Documentation engineering is a prime use-case for this technique. Tooling for the LSST Science Pipelines documentation is packaged in Documenteer. We can’t install Documenteer with the Pipelines’s EUPS stack, so instead we install Documenteer on top of the Pipelines’s EUPS stack with pip.

The method

I’m going to show you how to use venv to create the virtual environment because it’s included in the Python 3 standard library.

Before starting, you need to be working in a shell with the LSST environment loaded (loadLSST.bash) and set up (setup lsst_distrib).

Then create the virtual environment and pip-install packages:

python -m venv --system-site-packages --without-pip .pyvenv
source .pyvenv/bin/activate
curl https://bootstrap.pypa.io/get-pip.py | python
.pyvenv/bin/pip install -r requirements.txt

If you look inside the virtual environment, you should see your packages installed there:

ls .pyvenv/lib/python3.6/site-packages

At the same time, you should be able to import lsst from the parent miniconda Python:

python
>>> import lsst.afw
>>> lsst.afw.__version__

Notes

  • The --system-site-packages argument allows this virtual environment (.pyvenv) to see packages installed in the miniconda site-packages. For LSST Science Pipelines, miniconda is the “site.”
  • The --without-pip is needed because venv can’t bootstrap its own pip. This is because ensurepip is not available in miniconda. The work-around is to create the environment, then add pip later (tip via https://github.com/ContinuumIO/anaconda-issues/issues/6917#issuecomment-340014721).
  • .pyvenv/bin/pip install -r requirements.txt can be any pip installation command you require. Note that I did find it necessary to directly specify .pyvenv's pip because it wasn’t already at the top of the path, despite the environment being activated. If you have a fix for this, let me now.
  • .pyvenv is the name of the virtual environment, and also the directory containing the virtual environment. You can name it whatever you like. Let me know if there’s a better canonical name.
1 Like