I’m building a Jupyter Notebook with the LSST software added, and my Dockerfile reads:
RUN apt-get update && \
apt-get upgrade -yq --no-install-recommends \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir /opt/lsst && chown $NB_USER /opt/lsst
# python3 updates and additional files
RUN conda install --quiet --yes \
'xlrd=1.1.0' && \
conda clean -tipsy
# For some reason, I need to install these AFTER installing the stuff above
RUN conda install --quiet --yes \
&& conda remove --quiet --yes --force qt pyqt && \
conda clean -tipsy
# Now install
# Configure environment
ENV LC_ALL=en_US.UTF-8 \
RUN curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh && \
chmod +x newinstall.sh && \
bash ./newinstall.sh -b -P /opt/conda/bin/python && \
bash -c "source /opt/lsst/loadLSST.bash ; eups distrib install lsst_distrib; eups distrib install sims_maf -t sims"
(The Jupyter notebook is Ubuntu 16:04)
The conda install definitely adds ‘pycosat’:
pycosat-0.6.3 | py36_0 201 KB conda-forge
… which is an upgrade:
pycosat: 0.6.3-py36h0a5515d_0 defaults --> 0.6.3-py36_0 conda-forge
… and the old version is removed:
When it comes to the LSST install, it wants to reinstall pycosat-0.6.3-py36h0a5515d_0… and then barfs:
UnsatisfiableError: The following specifications were found to be in conflict:
- conda -> pycosat[version='>=0.6.3']
Use "conda info <package>" to see the dependencies for each package.
If I drop back to the 14.0 version… I’m fine.
We don’t need
pycosat as such. It’s pulled in by another package that we do need. We force explicit versions when building our system standalone in the test environment so that we know exactly what we are building against. The packages we actually require are: numpy scipy matplotlib requests cython sqlalchemy astropy pandas numexpr bottleneck h5py pyyaml (and
nomkl on linux). This is what you get if you set up a conda LSST environment using the
-b flag for
bin/deploy. We do build docker images already if you want to base your system on our image. We also have JupyterLab deployments under test.
Our Notebooks service is not moving to 'Labs for a while - and now, probably not until the 2019/20 academic year (academic teaching environment).
The challenge with jupyter-anything is that one needs to be able to launch a notebook-server (from JupyterHub) which starts a pre-configured web-GUI… and not start a terminal, do some config, and then start a notebook-server.
I’ve tried all sorts of things, and it’s just not playing ball… is there an assumption that the underlying OS will be redhat? The Jupyter notebooks are built on top of Ubuntu.
I’ve tried pre-loading the libraries Tim mentioned, I’ve tried down-grading the pycosat library, and I’ve tried forking the Jupyter notebook-stack and building on this-weeks new-version of ubuntu… none of them work.
My next one will be to try building the whole Notebook stack on top of the LSST Docker image (but that seems messy to me)
Have you got an account on lsst-dev?
If so, maybe you can try https://lsst-lspdev.ncsa.illinois.edu/nb to get to our JupyterLab environment , and then see what’s missing in the provided LSST Stack Python environment, and then open a terminal and pip install --user the things you need?
Implementing Jupyter on top of the stack is nontrivial.
At the least, doing it this way would tell you what you actually need in addition to what’s in the stack.
We make images available with the stack already on - you might want to start with one of those? https://pipelines.lsst.io/install/docker.html
@frossie - you do, indeed, provide up-to-date docker images with the full LSST stack on it… however to start a Notebook-Server you need to start the container, run some commands, and then start the 'server - this makes it hard to start from JupyterHub.
You might want to look at https://github.com/lsst-sqre/jupyterlabdemo/tree/master/jupyterlab for inspiration. There’s no reason you couldn’t start with something like that but set it up to fire up a classic Notebook server rather than the Lab (admittedly, you may not need the user provisioning fanciness–I don’t know what your persistent storage model looks like).
Fundamentally it’s a matter of adding the necessary packages on top of the stack container and then arranging for your entrypoint to do whatever user setup you need before starting the server that the Hub connects to.
@adam - I’ve seen this before (plus https://lsst-uk.atlassian.net/wiki/spaces/HOME/pages/141328388/LSST+Simulation+Docker+Image and Installing lsst_sims failed inside docker)
I’ll have another look (and yes, I could add all the notebook server stuff on-top of the LSST docker, rather than add the LSST stack on-top is a notebook docker)
(edit… also found this: https://hub.docker.com/r/oboberg/maf/)
The problem is that
newinstall.sh has no option to install cutting edge versions of packages. It’s been written solely to force specific versions. This is not true for the
lsstsw build system we use to build from
git which has an option to use whatever are the current versions. I think you may be able to put in a dot file in your system to con
newinstall.sh into thinking that it’s done the conda installs already, although I can’t remember what the file is. @josh ?
newinstall.sh does not use lockfiles.
Ok, so doing the newinstall and then forcing updates of all the packages should work fine.
Can someone explain the
eups distrib install bit - is it installing python packages, or lsst libraries?
I’ve got past
pycostat problem from
newinstall.sh and now butting heads on
- if I try & install 15_0, it wants to install 200+ items, of which
doxygen 1.8.5.lsst1 is one - which fails with an ICONV error
- if I try & install 14_0, it wants to install 100+ items, of which
doxygen 1.8.5.lsst1 is one - which installs just fine.
(I’m building on a Jupyter Notebook base, which is Ubuntu 16:04 [xenial] - however I’ve tried rebuilding their notebooks based on Ubuntu 18:04 [Bionic]… and both fail)