Running DM Jupyter Notebooks locally

Data Management
#1

Recently I wanted to run the tutorial notebooks on my laptop - it seems a few others also wanted to do this. The obvious approach is to use a docker container but the DM CentOS containers are minimal and have no Jupyter. The JupyterLab containers however do contain all you need to run notebooks and are the exact containers deployed when you login into the Jupyter environment at NCSA.

For this you need docker installed and running.

Of course you may need some data depending on the notebook - intro-process-ccd for example requires ci_hsc_small . The contents of this should be in a WORK directory (mine was in /Users/womullan/LSSTgit/notebooks/WORK).

Next you need some notebooks. I got the demo-notebooks and put them in /Users/womullan/LSSTgit/notebooks/.

Running the notebook environment with the container can then be done with a single command (where you substitute the womullan paths for your local directories):

docker run --volume=/Users/womullan/LSSTgit/notebooks:/tmp/notebooks \
  --volume=/Users/womullan/LSSTgit/notebooks/WORK:/tmp/WORK \
  -p 22000:22000 --rm -it \
  lsstsqre/sciplat-lab:recommended \
  /opt/lsst/software/stack/python/miniconda3-4.7.12/envs/lsst-scipipe-984c9f7/bin/jupyter \
    notebook --ip=0.0.0.0 --port=22000 --allow-root

This will output a URL to hit the notebook something like:
http://(cf6af7b52148 or 127.0.0.1):22000/?token=51f74a5d780d6e8f983eb93f413e9e69bfdd90396b93e792
Which means you should put :
http://127.0.0.1:22000/?token=51f74a5d780d6e8f983eb93f413e9e69bfdd90396b93e792
in the browser.

That is it.

Ancillary docker info :
If you want to connect ot a running container i.e the one you started above
find the container id with

docker ps

then connect with

docker exec -it  CONTAINERID /bin/bash

That allows you to pip install any other packages you want for example.

Running Jupyter notebooks under docker
#2

@womullan Is it possible to get these instructions updated? I am interested in running the stack on my laptop (with Docker) and having the Jupyter environment available. For example, I don’t think the jld-lab container exists anymore.

#3

Using lsstsqre/sciplat-lab:recommended instead of lsstsqre/jld-lab:d20180911 should work, along with the path /opt/lsst/software/stack/python/miniconda3-4.7.12/envs/lsst-scipipe-984c9f7/bin/jupyter to Jupyter. Keep in mind that the container image is currently 17.4GB in size and that the versions of miniconda and the lsst-scipipe conda environment change from time to time.

#4

Thanks @ktl! This worked perfectly. Are the dockerfiles that were used to build these images publicly available? I would be interested adding some additional functionality, Source Extractor, for example to this.

#5

You should be able to add your own layers on top of the lsstsqre/sciplat-lab containers by simply using a FROM directive at the beginning of your own Dockerfile.

But if you really want to look at how the sausage is made, a template is in https://github.com/lsst-sqre/nublado/blob/master/jupyterlab/Dockerfile.template
It uses an underlying Science Pipelines “stack” container itself. Caveat: we reserve the right to change this at any time.