Analyzing LSST DESC Twinkles data at NERSC via remote jupyter notebooks

drphilmarshall · January 9, 2017, 7:26pm

Thought you might like to know that Bryce ( @jbkalmbach ) is now analyzing our LSST DESC Twinkles data at NERSC via a remote jupyter notebook, using NERSC’s experimental “jupyter-dev” system. Here are his notes:

github.com

LSSTDESC/Monitor/blob/master/doc/jupyter-dev.md

# Monitor @ NERSC: Introducing `jupyter-dev`

## What is `jupyter-dev`?
  `Jupyter-Dev` is a system set up at NERSC to run jupyter notebooks on the cori filesystem by simply pointing your web browser to https://jupyter-dev.nersc.gov and using your NERSC credentials. It does take a bit of setup before the first time, but while the jupyter-hub system runs on a server external to Cori the `jupyter-dev` system has access to the Cori filesystem. Thus, it can be configured to run custom kernels set up in your home directory. This allows a user to create LSST stack enabled kernels where LSST software as well as any additional packages required are accessible in a `jupyter` notebook.
  
## Why bother with `jupyter-dev`?
We want to use `jupyter-dev` for our project in order to have a convenient portal to the [Twinkles](https://github.com/LSSTDESC/Twinkles/blob/master/README.md) outputs stored in science databases at NERSC. It also gives us a common platform in which to run analysis. The directions below point all users to run a shared version of the LSST stack. Therefore, any notebooks we create in `jupyter-dev` can be easily shared and will work for others using the shared stack in their own `jupyter-dev` browser window.

## Steps for setting up `jupyter-dev` for the Monitor

1. #### SSH in to Cori

  Before you are able to run `jupyter-dev` with an lsst kernel you'll need to do some setup on Cori.

2. #### Load the anaconda module to get ipython

  Once logged into Cori type: `module load python/2.7-anaconda`

3. #### Setup Kernel Spec

This file has been truncated. show original

It’s easy for DESC members to get NERSC accounts, so if you’re interested in experimenting with working in this SUI-like mode, let us know.

Gregory (@gpdf), Xiuqin (@xiuqin), Trey (@roby), David (@davidciardi): are there any cool SUIT widgets you think we should be using already?

mgckind · January 9, 2017, 9:31pm

@drphilmarshall, This is from last week, https://arxiv.org/abs/1701.01222 and it might be of interest. Its a jupyter-based slippy-map interactive extension tool for catalog data with different layers overlay.

drphilmarshall · January 12, 2017, 7:53pm

Thanks @mgckind! We’ll try and check Vizic out