LSST Stack 'on Ubuntu Linux on Windows'

Data Management
, ,
#1

Original (04/06/16), Insider Preview Build 14316:

This summer Microsoft is releasing their anniversary update to Windows 10 which includes their “Windows Subsystem for Linux” allowing the user to run Ubuntu Linux from the Windows command prompt. The developer insider preview build was released this morning, so inevitably…

… I tried to install the LSST stack on Windows.

Here are my notes of the attempt:

  • I’ve successfully installed the windows preview build. (I had to remove my graphics drivers in safe mode, although I suspect I could just have disabled SLI before the windows update.)
  • The “Windows Subsystem for Linux” works.
  • I was unable to install all the pre-req libraries.
  • I was unable to install the anaconda python distribution due to a symlink issue.
  • I was unable to install the LSST stack from source, I tried with the miniconda but it also failed citing symlink issues. Without the miniconda it managed to install EUPS but failed at creating the basic environment citing symlink issues.

Sample symlink errors:
tar: addon/doxmlparser/src/doxmlintf.h: Cannot create symlink to ‘…/include/doxmlintf.h’: Invalid argument
tar: bin/python: Cannot create symlink to ‘python2.7’: Invalid argument
tar: bin/python2: Cannot create symlink to ‘python2.7’: Invalid argument
tar: lib/libpython2.7.so: Cannot create symlink to ‘libpython2.7.so.1.0’: Invalid argument
tar: lib/pkgconfig/python.pc: Cannot create symlink to ‘python-2.7.pc’: Invalid argument
tar: lib/pkgconfig/python2.pc: Cannot create symlink to ‘python-2.7.pc’: Invalid argument

So it seems from a very preliminary attempt and a bit of Googling that Microsoft currently needs to reconcile differences between file storage systems on Windows and the Linux subsystem allowing symlinks to work completely.

Update (05/24/16), Insider Preview Build 14342:

Microsoft released a few new features and fixes for the Windows Subsystem for Linux (WSL) including support for symlinks across different file systems. Time for attempt number two.

Notes:

  • Managed to install Anaconda.
  • Did not succeed in installing all the pre-req software.
  • Managed to install the conda distributed LSST stack.
  • Managed to install the stack from source.
  • However, due to missing pre-req dependencies the stack in both cases did not work. (Attempted importing lsst.afw.image in python to no avail).

Currently, WSL does not have the /proc filesystem fully implemented stopping libcurl4-openssl-dev, openjdk-7-jre and ca-certificates from apt-get installing. Fortunately, the issue is being tracked by Microsoft developers and has been fixed with a possible release within a few weeks. Full disclosure: there is an installation workaround of jdk / jre, however I have not tried it (…yet?).

All in all, more progress, and more importantly, progress in the right direction. Another update when the fix is live!

Update (05/26/16), Insider Preview Build 14352:

Microsoft deployed the /proc fix and now all listed pre-requisite libraries install from apt-get. Unfortunately, there are still unresolved errors. See comments and ongoing discussion below.

Links for the curious:
Insider Preview Blog (with Release Notes)
Community Feedback Forum
Windows Subsystem for Linux Release Notes
Windows Subsystem for Linux GitHub

2 Likes
Installing Science Pipeline for LSST@Europe conf. tuturials, issue with
#2

I applaud your bravery.

1 Like
#3

Huh. Well, that’s progress!

1 Like
#4

@RHL must have really mixed feelings about this :slight_smile: .

1 Like
#5

Btw., I just noticed:

Pasted image1054×130 69.4 KB

:laughing: :laughing: :laughing: :laughing:

PS: On a more serious note, this could be quite cool. I’m now actually mildly excited about Windows (for the first time since ~NT 4…). Thanks for trying this @moeyensj.

1 Like
#6

It was meant to be. :smiling_imp:

I’ll stay up to date with the latest updates from Microsoft and report back in the near(ish) future.

#7

Attempt two notes added to original post.

#8

Curious as to what is failing on import of lsst.afw.image; I wouldn’t think that libcurl should be needed, and jdk should only be needed by qserv.

#9

Hi @ktl, Microsoft just released a new build and I already updated so I can’t reproduce the conditions 100%, however, I still get the same error:

ImportError: libssl.so.10: cannot open shared object file: No such file or directory.

libssl-dev seems to depend on ca-certificates which couldn’t be apt-get installed. Now, according to the release notes for the new build released last night, Microsoft may have fixed what was preventing that from installing. I will give it a try.

#10

is this in one of the daf packages? If so it may be that you are suffering from the mariadbclient problem of discovering a different libssl from the standard one.

#11

Wonder if it’s a manifestation of https://github.com/mjuric/conda-lsst/issues/64? Maybe try symlinking libssl.so.1.0.0 (or whichever libssl is present) to libssl.so.10.

#12

@mjuric @timj Apologies for the delay! Symlinking libssl.so.1.0.0 to libssl.so.10 worked to fix the ImportError (especially now that all the stack pre-requisite software installs from apt-get without difficulty). It then continues to generate an ImportError for libcrypto.so.10:

ImportError: libcrypto.so.10: cannot open shared object file: No such file or directory

However, symlinking libcrypto.so.1.0.0 (or any other version) to libcrypto.so.10 doesn’t work and produces the following error:

ImportError: /lib/x86_64-linux-gnu/libcrypto.so.10: version `libcrypto.so.10' not found (required by /home/moeyensj/anaconda2/opt/lsst/mariadbclient/lib/libmysqlclient.so.18)

There was some indication online that forcing a reinstall of libssl.1.0.0 followed by a symlink might fix the issue (ie: sudo apt-get install libssl1.0.0 --reinstall), but it didn’t work. To my understanding libssl1.0.0 is the only package that provides libcrypto.so.1.0.0 so I am unsure of how to proceed from here.

#13

Hi @moeyensj,

I have the same problem when I try to execute processCcd.py from a docker ubuntu image. Despite the creation of the following symlinks:

RUN ln -s /lib/x86_64-linux-gnu/libssl.so.1.0.0 /usr/lib/libssl.so.10       && \
    ln -s /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /usr/lib/libcrypto.so.10

it is ok for /usr/lib/libssl.so.10 but it does not work for /usr/lib/libcrypto.so.10.

On the the other hand, I do not need to create those symlinks with a centos image (it works “out-of-the-box”).

#14

Hi @YvanCalas, I just managed to import lsst.afw.image in Python without it complaining (@mjuric @timj). Hopefully, this will fix it for you as well.

Assuming you have the following symlinks setup:

sudo ln -s /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libssl.so.10```

In order to fix this error:
```ImportError: /lib/x86_64-linux-gnu/libcrypto.so.10: version `libcrypto.so.10' not found (required by /home/moeyensj/anaconda2/opt/lsst/mariadbclient/lib/libmysqlclient.so.18)```

I installed a new libcrypto and then symlinked to that:
```sudo apt-get install libcrypto++
sudo rm -rf /lib/x86_64-linux-gnu/libcrypto.so.10
sudo ln -s /lib/x86_64-linux-gnu/libcrypto++.so.9.0.0 /lib/x86_64-linux-gnu/libcrypto.so.10```

At this stage, trying to `import lsst.afw.image` in Python then throws the following error:
```ImportError: /lib/x86_64-linux-gnu/libssl.so.10: version `libssl.so.10' not found (required by /home/moeyensj/anaconda2/opt/lsst/mariadbclient/lib/libmysqlclient.so.18)```

In a similar fashion to libcrypto, I symlinked to a newer version of libssl:
```sudo rm -rf /lib/x86_64-linux-gnu/libssl.so.10
sudo ln -s /lib/x86_64-linux-gnu/libss.so.2 /lib/x86_64-linux-gnu/libssl.so.10```

And now it looks like the LSST stack works on the Windows Subsystem for Linux to some capacity.
#15

Since this is the mariadbclient issue, I think the fix is to build mariadbclient using the internal SSL library as is already done on OS X. There have been many discussions on this topic in JIRA such as DM-5802 and DM-5170. Maybe the test for use of bundled SSL could be expanded for this platform.

#16

Thanks a lot @moeyensj for the hint. Unfortunately it does not solve my issue. I now have another error message:

Execution du processCcd.py Traceback (most recent call last): File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/pipe_tasks/2016_01.0-30-g4f51cf7/bin/processCcd.py", line 23, in <module> from lsst.pipe.tasks.processCcd import ProcessCcdTask File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/pipe_tasks/2016_01.0-30-g4f51cf7/python/lsst/pipe/tasks/processCcd.py", line 22, in <module> from lsst.ip.isr import IsrTask File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/ip_isr/2016_01.0-7-g806b452+8/python/lsst/ip/isr/__init__.py", line 24, in <module> from .isrLib import * File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/ip_isr/2016_01.0-7-g806b452+8/python/lsst/ip/isr/isrLib.py", line 34, in <module> _isrLib = swig_import_helper() File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/ip_isr/2016_01.0-7-g806b452+8/python/lsst/ip/isr/isrLib.py", line 30, in swig_import_helper _mod = imp.load_module('_isrLib', fp, pathname, description) File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/base/2016_01.0+5/python/lsstimport.py", line 102, in imp_load_module module = orig_imp_load_module(name, *args) ImportError: /sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/mariadbclient/10.1.11.lsst2/lib/libmysqlclient.so.18: symbol SSL_accept, version libssl.so.10 not defined in file libssl.so.10 with link time reference

From the ubuntu:latest docker image, the following commands are executed:

RUN apt-get update && apt-get -y install perl python libssl1.0.0 libcrypto++ libssl-dev
RUN ln -s /lib/x86_64-linux-gnu/libss.so.2 /lib/x86_64-linux-gnu/libssl.so.10  && \
    ln -s /usr/lib/libcrypto++.so.9.0.0 /lib/x86_64-linux-gnu/libcrypto.so.10

Could it a bad compiled version of the stack compiled at CC-IN2P3?

#17

My guess is that this is fall out from soft linking the library, and the symbols in the libssl that Mariadbclient found initially are not the same as the .10 version that you faked with the soft link.

The real fix is for mariadbclient to use the bundled SSL.

#18

+1. @frossie, is there time for this to still make it into 12.0 (I think it’s a small change in mariadb(client)'s eupspkg.cfg.sh), or is it too late and/or risky?

PS: conda packages could be built against conda's build of OpenSSL (but it’s not required).

#19

I think conda builds probably should use the conda ssl and that’s what mariadbclient finds. The problem is that nothing in our default non-conda builds includes the conda libraries in the LD_LIBRARY_PATH so they can not be found later on.

I think the objection to using the bundled SSL is that the hope was the system ssl would be easier to keep up to date.

We have multiple test branches for mariadbclient that attempt to deal with this problem of cmake being too clever at finding libraries (using ../lib in each PATH entry).

#20

Epic :slight_smile: .

Now for a real challenge ;-), I wonder if any of the notebooks from sims_maf_contrib would work (e.g., the Introduction Notebook). Send proof in form of the notebook running in MS Edge with today’s copy of the New York Times opened in a different tab :slight_smile: .