Using an lsst-dev stack w/ local ipython & ds9 tutorial

Many people’s workflows these days involve running IPython notebooks, and often ds9 as well as their image browser of choice. However, these don’t play all that nicely with running the stack on lsst-dev, as IPython notebook are controlled through web-browsers, and using ds9 with X-forwarding largely negates the advantages offered by ds9 (fast, interactive re-scaling & panning etc.)

This is a tutorial designed to help people setup and run the stack on lsst-dev, with the IPython kernels running there, but controlling these in a local web-browser, and with a local copy of ds9, as opposed running ds9 on lsst-dev and X-forwarding the images (and all the lag that brings).

On your local machine, create/edit the file: ~/.ssh/config and add the entry:

Host lsst-dev
    Hostname lsst-dev.ncsa.illinois.edu
    User <your username>
    RemoteForward XXXX localhost:XXXX
    RemoteForward <XXXX+1> localhost:<XXXX+1>
    LocalForward YYYY localhost:ZZZZ 
    ForwardAgent yes
    ForwardX11 yes
    ForwardX11Trusted yes

where XXXX and XXXX+1 are random consecutive numbers greater than 4096, and YYYY and ZZZZ are any other random numbers greater than 4096. I have not included values here, because otherwise once multiple people have followed these instructions we will end up with a bunch of collisions and a tangled, buggy mess!

On your local machine configure the XPA ports for ds9 by typing:

export XPA_PORT="DS9:ds9 XXXXX XXXXX+1"

and open an ssh tunnel with port forwarding:

ssh -N -f -L localhost:ZZZZ:localhost:YYYY username@lsst-dev.ncsa.illinois.edu

using the values from above. Now, launch ds9. If you have a from-source version then just launch by typing ‘ds9 &’ from the same terminal. If you have an OS-X “app style” distribution you can try launching it from the OS-X GUI as you usually would, but you will need check whether it has picked up the XPA port config from the environment by clicking File—>XPA—>Information in ds9 and confirming that the number after the colon under XPA_METHOD is equal to XXXXX from before (sometimes it is, sometimes it isn’t, I haven’t figured out why :confused: ). If it is then you’re fine. If it’s not then close ds9 and relaunch the ‘app’ from the terminal by typing:

open /Applications/SAOImage\ DS9.app/ &

Now, login to lsst-dev by typing just:

ssh lsst-dev

so that it picks up the config from the ssh config file. You should not be prompted for a password (unless it is for your ssh key via an OS-X dialog) as you should have set up your keys. I believe that if this is not the case then you are going to have problems.

On lsst-dev setup the stack: (or use one of these methods)

source /lsst/stack/loadLSST.sh (or appropriate for your shell)
setup lsst_apps

and specify the XPA ports on this machine too, using the same values as before:

export XPA_PORT="DS9:ds9 XXXXX XXXXX+1”

Now, cd to wherever your IPython notebook directory is, eg:

cd ~/ipython_notebooks

and launch the IPython notebook with no browser, specifying the port from before:

ipython notebook --no-browser --port=YYYY

or, for a continuous session something along the lines of:

nohup ipython notebook --no-browser --port=YYYY >> ~/ipython_notebooks/stdout.txt

Now, on your local machine, launch firefox (or your browser of choice), and go to:

localhost:ZZZZ

This should connect you to the remote kernel. To test that you have a remote kernel with the stack loaded correctly and the instance of ds9 on your local machine correctly connected, open a new notebook and do the following:

import lsst.afw.image as afwImage
import lsst.afw.display as afwDisplay
import numpy as np

data = np.zeros((10,10), dtype=np.float32)
img = afwImage.ImageF(data)
afwDisplay.mtv(img,frame=0)

a small, 10x10 black box should appear inside ds9 on your local machine.

If it does, you have succeeded! :smiley:
If it does not, I have failed :weary:

PS: If you just want to try a local stack with a local ds9:

Open ds9 (either cmdline version or the “App”), mainly just to check if you have it installed. The stack should try to launch it for you, but I think this only works for the cmdline version and not the App.

import lsst.afw.display.ds9 as ds9
import lsst.afw.image as afwImage
import numpy as np
data = np.zeros((100,100), dtype = np.float32)
img = afwImage.ImageF(data)
ds9.mtv(img,frame=1)
5 Likes

A few further points worth noting:

Due to current bugs with the stack and its setup, certain problems are likely to be encountered when following these instructions. Firstly, IPython is most likely not installed for a given user, so a local copy will need to be installed as it no longer ships with the stack. This, however, will often cause its own problems.

I have also encountered several problems with cloning, tagging and building my own versions of parts of the stack, so here is a recipe for building a custom stack, including building on lsst-dev, and installing correct versions of IPython etc so that the tutorial above works.

Before doing anything:
pip uninstall --user ipython pip uninstall --user jupyter

To setup a shared stack:
. ~lsstsw/bin/setup.sh setup -t b1946 lsst_apps setup -t b1946 -r ~/lsst/obs_decam

add ~/.local/bin to $PATH (each time, so put this in you .bash_profile)

To clone obs_decam (or a python-only repo) just:
mkdir ~/lsst cd ~/lsst git clone git://github.com/lsst/obs_decam.git

To clone afw (or a repo which needs building):
cd ~/lsst git clone git://github.com/lsst/afw.git

To build afw:
For this the stack must be setup, so need to at least run:
. ~lsstsw/bin/setup.sh setup -t b1946 lsst_apps
and then, to build
exec scl enable devtoolset-3 bash ## must enable other compiler to build
setup -j afw -r ~/lsst/afw/ cd ~/lsst/afw scons -Q opt=3 -j 6

Now reinstall packages:
pip install --user ipython pip install --user jupyter

I hope that the magical incantations contained in this post are an approximately ordered superset of those needed to make everything work.

Finally, as someone who is neither in camp Emacs nor camp Vi, but who likes to use an IDE when editing code (god forbid!), one final nice touch to my workflow I found which I thought I would share here is using sshfs.

Essentially, one uses a local IDE of their choice, and then remotely mounts the relevant portion of the lsst-dev filesystem locally.

For example, I run a local version of obs_decam and afw (as per the tutorial) which live in ~/lsst/obs_decam/ and ~/lsst/afw/ respectively. When I want to edit code in these packages I simply type

sshfs username@lsst-dev.ncsa.illinois.edu:/home/username/lsst ~/lsst-dev

and then open PyCharm on my Mac and work live on the files without having to run annoying rsyncs or similar in order to push changes to my stack on lsst-dev. I find that this not only removes a significant amount of friction from my workflow, but also removes the possibility of editing code locally but forgetting to push the changes over and the frustration which results from debugging that cycle.

If anyone has similar workflow hints & tips then I would love to hear them!

If you have a local machine with an X server (Linux or Mac), then you might want to try using xpra, which has been installed on lsst-dev. You’ll need to install it on your local machine as well; it’s probably available from whatever package manager you use, but if not then you can get it from http://www.xpra.org .

xpra can be thought of as “screen for X”. It is much faster than a regular X connection when you don’t have a lot of bandwidth (e.g., working remotely), and it saves state between connections. I find it simpler and better than VNC (e.g., you’re not constrained by the remote X display manager, which is usually pretty basic). Here’s how to use it:

Choose a DISPLAY number that’s unused by anyone else, starting at 10 (but don’t use 10 because that’s mine…). We’ll choose 11 for the sake of this demo.

On lsst-dev:

xpra start :11
export DISPLAY=:11

On your machine:

xpra attach ssh:lsst-dev:11

Leave that running (or you could put it in the background, and use xpra detach :11 later).

You should put the full address for lsst-dev and your login name if you don’t have it specified in your ~/.ssh/config file.

Then you can open windows on lsst-dev (with DISPLAY=:11) and they will appear on your machine. If you now kill the xpra attach on your machine, you’ll lose those windows. When you reattach, they’ll reappear.

2 Likes

Hi Merlin,

Have you used this on the new lsst-dev machine? I followed this (very helpful, thanks) post before the switch over with no problem but now I’m getting an error from jupyter notebook:

socket.error: [Errno 99] Cannot assign requested address

Have you seen this before?

-Bryce

Try:

$ ipython notebook --no-browser --ip=127.0.0.1 --port=YYYY

Seems to work for me.

Credit here: https://github.com/ipython/ipython/issues/6193#issuecomment-50062534

Thanks for the answer @swinbank, and @jbkalmbach, please let us know if that works for you too! :slight_smile:

Yep, that did it. Thanks a lot @swinbank!

You don’t need to specify a port for this magic to work; as usual jupyter will choose an available port >= 8888. You can say --port YYYY if you like.

This still seems to be necessary, and John’s GitHub link implies that it’s something due to a misconfiguration problem at NCSA. Can someone take a look? I’m pinging @daues, but I’ve no idea if he’s the right person.

The problem is that lsst-dev01 does not recognize localhost as 127.0.0.1, despite it being in /etc/hosts. I’m not sure if this is intentional or a misconfiguration.

I’ll gather the response of our admins to this. Some searching around suggests the order of entries in /etc/hosts may be wrong (?)
https://bugzilla.redhat.com/show_bug.cgi?id=1093037

I may have been wrong about this (unless something changed). ping localhost works fine. I was using a different (probably incorrect) tool.

I think the issue is that localhost is both 127.0.0.1 and ::1. IPython seems to be trying to bind to the latter rather than the former, and that’s what fails.

I gave this a try today and @Merlin was very kind to help over Slack! Ultimately, I wasn’t able to get it working, because no local copy of ds9 I launched was ever willing to grab the port numbers I configured with XPA_PORT. It would instead insist on choosing something like XPA_METHOD: /var/folders/gh/wk3w_d9d5qx_550341rf7gmh0000gn/T//DS9_ds9.45817 (where the last number is apparently randomly assigned).

This turned out to be caused by the line export XPA_METHOD=local in my .bashrc, which I added ages ago to stop ds9 from often telling me “XPA unable to verify hostname, setting XPA_METHOD to LOCAL” in a pop-up when it launched. With the XPA_LOCAL export removed, and the XPA_PORT export in place, ds9 apparently tries to take the XPA_PORT assignment and fails because it doesn’t believe my hostname can be verified. It displays the popup warning yet again and does not set XPA_METHOD as desired.

This problem is apparently indicative of how one’s internet hostname is managed(?), and not something I can easily work around. If you know otherwise, I’d love to hear more!

1 Like

I strongly recommend using xpra instead of fiddling with xpa. It’s much easier.

1 Like

Thanks @price, that is indeed easier (once I installed xpra on my Mac). The ds9 window has very weird/ugly fonts when it loads over xpra, but it appears reasonably promptly, and that’s essentially what I was after!

I think the fonts issue clears up if you reset the ds9 preferences, or you explicitly set a font size, or something like that.

I am not sure if this is relevant. But I cannot make this work with the current stacks available on lsst-dev (I actually tried w_2017_40, v14_0 and w_2017_44). The failure mode is that ipython and claims to run fine in the terminal. But it does not show up on my local browser (I get the default THIS WEBSITE CANNOT BE REACHED). I went through the checklist in Ds9 communication via XPA through step 6 to no avail. I talk with a Sam Thrush who had previously made it work with lsst_distrib -t w_2017_18. But that is no longer an option on lsst-dev, and she gets the same failure mode as I do now. Is there an available stack that makes this work?

Sorry, this isn’t really debug information, but this isn’t a problem with the stack version as I just tried w_44 on lsst-dev, and have no problems connection to either the ipython kernel, nor forwarding data to ds9.

It sounds like your ssh tunnels aren’t working/configured correctly; if I try to connect to localhost:<random_number> I get that error too, so I suggest that something in the ssh and tunneling setup is the point of failure. Hope that’s of non-zero help.