is this in one of the daf packages? If so it may be that you are suffering from the mariadbclient problem of discovering a different libssl from the standard one.
Wonder if it’s a manifestation of https://github.com/mjuric/conda-lsst/issues/64? Maybe try symlinking libssl.so.1.0.0 (or whichever libssl is present) to libssl.so.10.
@mjuric @timj Apologies for the delay! Symlinking libssl.so.1.0.0 to libssl.so.10 worked to fix the ImportError (especially now that all the stack pre-requisite software installs from apt-get without difficulty). It then continues to generate an ImportError for libcrypto.so.10:
ImportError: libcrypto.so.10: cannot open shared object file: No such file or directory
However, symlinking libcrypto.so.1.0.0 (or any other version) to libcrypto.so.10 doesn’t work and produces the following error:
ImportError: /lib/x86_64-linux-gnu/libcrypto.so.10: version `libcrypto.so.10' not found (required by /home/moeyensj/anaconda2/opt/lsst/mariadbclient/lib/libmysqlclient.so.18)
There was some indication online that forcing a reinstall of libssl.1.0.0 followed by a symlink might fix the issue (ie: sudo apt-get install libssl1.0.0 --reinstall), but it didn’t work. To my understanding libssl1.0.0 is the only package that provides libcrypto.so.1.0.0 so I am unsure of how to proceed from here.
Hi @moeyensj,
I have the same problem when I try to execute processCcd.py from a docker ubuntu image. Despite the creation of the following symlinks:
RUN ln -s /lib/x86_64-linux-gnu/libssl.so.1.0.0 /usr/lib/libssl.so.10 && \
ln -s /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /usr/lib/libcrypto.so.10
it is ok for /usr/lib/libssl.so.10 but it does not work for /usr/lib/libcrypto.so.10.
On the the other hand, I do not need to create those symlinks with a centos image (it works “out-of-the-box”).
Hi @YvanCalas, I just managed to import lsst.afw.image in Python without it complaining (@mjuric @timj). Hopefully, this will fix it for you as well.
Assuming you have the following symlinks setup:
sudo ln -s /lib/x86_64-linux-gnu/libssl.so.1.0.0 /lib/x86_64-linux-gnu/libssl.so.10```
In order to fix this error:
```ImportError: /lib/x86_64-linux-gnu/libcrypto.so.10: version `libcrypto.so.10' not found (required by /home/moeyensj/anaconda2/opt/lsst/mariadbclient/lib/libmysqlclient.so.18)```
I installed a new libcrypto and then symlinked to that:
```sudo apt-get install libcrypto++
sudo rm -rf /lib/x86_64-linux-gnu/libcrypto.so.10
sudo ln -s /lib/x86_64-linux-gnu/libcrypto++.so.9.0.0 /lib/x86_64-linux-gnu/libcrypto.so.10```
At this stage, trying to `import lsst.afw.image` in Python then throws the following error:
```ImportError: /lib/x86_64-linux-gnu/libssl.so.10: version `libssl.so.10' not found (required by /home/moeyensj/anaconda2/opt/lsst/mariadbclient/lib/libmysqlclient.so.18)```
In a similar fashion to libcrypto, I symlinked to a newer version of libssl:
```sudo rm -rf /lib/x86_64-linux-gnu/libssl.so.10
sudo ln -s /lib/x86_64-linux-gnu/libss.so.2 /lib/x86_64-linux-gnu/libssl.so.10```
And now it looks like the LSST stack works on the Windows Subsystem for Linux to some capacity.Since this is the mariadbclient issue, I think the fix is to build mariadbclient using the internal SSL library as is already done on OS X. There have been many discussions on this topic in JIRA such as DM-5802 and DM-5170. Maybe the test for use of bundled SSL could be expanded for this platform.
Thanks a lot @moeyensj for the hint. Unfortunately it does not solve my issue. I now have another error message:
Execution du processCcd.py Traceback (most recent call last): File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/pipe_tasks/2016_01.0-30-g4f51cf7/bin/processCcd.py", line 23, in <module> from lsst.pipe.tasks.processCcd import ProcessCcdTask File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/pipe_tasks/2016_01.0-30-g4f51cf7/python/lsst/pipe/tasks/processCcd.py", line 22, in <module> from lsst.ip.isr import IsrTask File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/ip_isr/2016_01.0-7-g806b452+8/python/lsst/ip/isr/__init__.py", line 24, in <module> from .isrLib import * File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/ip_isr/2016_01.0-7-g806b452+8/python/lsst/ip/isr/isrLib.py", line 34, in <module> _isrLib = swig_import_helper() File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/ip_isr/2016_01.0-7-g806b452+8/python/lsst/ip/isr/isrLib.py", line 30, in swig_import_helper _mod = imp.load_module('_isrLib', fp, pathname, description) File "/sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/base/2016_01.0+5/python/lsstimport.py", line 102, in imp_load_module module = orig_imp_load_module(name, *args) ImportError: /sps/lsst/dev/fabio/lsst_w_2016_15/Linux64/mariadbclient/10.1.11.lsst2/lib/libmysqlclient.so.18: symbol SSL_accept, version libssl.so.10 not defined in file libssl.so.10 with link time reference
From the ubuntu:latest docker image, the following commands are executed:
RUN apt-get update && apt-get -y install perl python libssl1.0.0 libcrypto++ libssl-dev
RUN ln -s /lib/x86_64-linux-gnu/libss.so.2 /lib/x86_64-linux-gnu/libssl.so.10 && \
ln -s /usr/lib/libcrypto++.so.9.0.0 /lib/x86_64-linux-gnu/libcrypto.so.10
Could it a bad compiled version of the stack compiled at CC-IN2P3?
My guess is that this is fall out from soft linking the library, and the symbols in the libssl that Mariadbclient found initially are not the same as the .10 version that you faked with the soft link.
The real fix is for mariadbclient to use the bundled SSL.
+1. @frossie, is there time for this to still make it into 12.0 (I think it’s a small change in mariadb(client)'s eupspkg.cfg.sh), or is it too late and/or risky?
PS: conda packages could be built against conda's build of OpenSSL (but it’s not required).
I think conda builds probably should use the conda ssl and that’s what mariadbclient finds. The problem is that nothing in our default non-conda builds includes the conda libraries in the LD_LIBRARY_PATH so they can not be found later on.
I think the objection to using the bundled SSL is that the hope was the system ssl would be easier to keep up to date.
We have multiple test branches for mariadbclient that attempt to deal with this problem of cmake being too clever at finding libraries (using ../lib in each PATH entry).
Epic 
.
Now for a real challenge ;-), I wonder if any of the notebooks from sims_maf_contrib would work (e.g., the Introduction Notebook). Send proof in form of the notebook running in MS Edge with today’s copy of the New York Times opened in a different tab .
@mjuric Well, I am not immune to the coolness (craziness?) factor here, but I’m pretty swamped and would have to do redo some testing so… [now watch for this stunning punting manouver down the left flank in slow mo…] if @timj can manage to do the honours succesfully tomorrow while I’m LDM-151ing, why the heck not.
Erm, and nobody tell @jbecla.
I can definitely make it so that mariadb and mariadbclient always use bundled libz and libssl. Is @josh going to object? Regardless, I can make the ticket and send it for review quick enough so that people can decide.
I’ve adjusted DM-5802 to use bundled SSL and ZLIB in mariadbclient. I’ll now make the identical change to mariadb.
Whilst I’m confident this will work fine (I’ve done a Jenkins build and OS X build). Who is going to check that this fixes the problem described here?
Yes. libssl averages around two vulnerabilities per month. Has anyone confirmed that the mariadb ssl symbols are not exported?
Jupyter notebooks don’t work with the current WSL build. (See: WSL Issue 185 and Jupyter Notebook Issue 1331) There is a potential workaround but the fix would be better if implemented by Microsoft developers. Fear not however, I shall accept your challenge when the time is right.
If someone was willing to guide me through the process I could and would be happy to test any fixes.
It doesn’t seem like we can resolve the MariaDB SSL issue in a timely manner that would allow it to squeeze into the release of 12.0. No one solution is liked by everyone (bundled SSL, stripping $PATH, adding conda library location to LD_LIBRARY_PATH).
For the record so far our options for fixing have been:
- Giving cmake an exclusion path.
- We currently try to ignore the library directory relative to the bin directory containing the python we are using. I’m not entirely sure why this isn’t working on ubuntu.
- This didn’t work when conda builds were done because there are two conda paths in the PATH with libssl. A patch to fix this and include both paths does exist but not pushed anywhere.
- Limiting the PATH available to cmake so that it won’t find the wrong libssl.
- This was thrown out as unreliable (and possibly wouldn’t even work on conda installs that have their own GCC in the conda path.
- Giving up and using the bundled SSL (and ZLIB).
- This will obviously fix everything with the caveat that you won’t get security patches without updating MariaDB.
- Lack of security patches is fine if we don’t ever use SSL in MariaDB but I don’t know enough about Qserv to know if that is right. @fritzm?
Of course, a case could be made that we should absolutely be using the conda libssl as it’s updated far more often than the system one. In that case we are approaching this from the wrong direction and should be working out how to put the conda library paths into LD_LIBRARY_PATH and how to get the build to use it explicitly.