Installing from source with newinstall.sh on RHEL 6: libstdc++.so.6: cannot open shared object file

josePhoenix · July 25, 2016, 8:54pm

I’m trying to install the LSST Stack using the newinstall.sh script for v12.0. I’m getting stuck on building mariadbclient, but I’m not sure how to diagnose the error. I’m using the RedHat software collections “devtoolset-3” to get a more recent gcc (4.9.2) per instructions here: C++11/14 (gcc 4.8) now the baseline.

  [  1/62 ]  cfitsio 3360.lsst4 ... done.
  [  2/62 ]  doxygen 1.8.5.lsst1 (already installed) done.
  [  3/62 ]  eigen 3.2.5.lsst1 ... done.
  [  4/62 ]  fftw 3.3.4.lsst2 ... done.
  [  5/62 ]  gsl 1.16.lsst3 ... done.
  [  6/62 ]  mariadbclient 10.1.11.lsst3 ... + setup --type=build -k -r .
++ /astro/ferguson1/jlong/lsststack/eups/bin/eups_setup DYLD_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib --type=build -k -r .
+ eval export 'MYSQL_INCLUDE_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/include/mysql;' export 'DYLD_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib;' export 'SETUP_MARIADBCLIENT='\''mariadbclient' LOCAL:/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3 -f Linux64 -Z '(none)'\'';' export 'LD_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib:/opt/rh/devtoolset-3/root/usr/lib64:/opt/rh/devtoolset-3/root/usr/lib:/opt/rh/rh-ruby22/root/usr/lib64;' export 'MARIADBCLIENT_DIR=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3;' export 'PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/bin:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/bin:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/bin:/astro/ferguson1/jlong/lsststack/Linux64/doxygen/1.8.5.lsst1/bin:/astro/ferguson1/jlong/lsststack/eups/bin:/astro/ferguson1/jlong/lsststack/Linux64/lsst/2016_01.0-8-g0fc2317+2/bin:/astro/ferguson1/jlong/lsststack/Linux64/scons/2.3.5/bin:/astro/ferguson1/jlong/lsststack/Linux64/sconsUtils/2016_01.0-4-g49bc714/bin:/astro/ferguson1/jlong/lsststack/Linux64/miniconda2/3.19.0.lsst4/bin:/user/jlong/miniconda3/bin:/user/jlong/.linuxbrew/bin:/opt/rh/devtoolset-3/root/usr/bin:/opt/rh/rh-ruby22/root/usr/bin:/opt/rh/git19/root/usr/bin:/home/jlong/bin:/usr/local/bin:/opt/rh/rh-java-common/root/usr/bin:/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/grp/software/Linux/Wolfram/Mathematica/9.0/bin:/grp/software/Linux/RH6/x86_64/sm-2.4.36/bin;' export LSST_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib
++ export MYSQL_INCLUDE_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/include/mysql
++ MYSQL_INCLUDE_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/include/mysql
++ export DYLD_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib
++ DYLD_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib
++ export 'SETUP_MARIADBCLIENT=mariadbclient LOCAL:/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3 -f Linux64 -Z (none)'
++ SETUP_MARIADBCLIENT='mariadbclient LOCAL:/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3 -f Linux64 -Z (none)'
++ export LD_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib:/opt/rh/devtoolset-3/root/usr/lib64:/opt/rh/devtoolset-3/root/usr/lib:/opt/rh/rh-ruby22/root/usr/lib64
++ LD_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib:/opt/rh/devtoolset-3/root/usr/lib64:/opt/rh/devtoolset-3/root/usr/lib:/opt/rh/rh-ruby22/root/usr/lib64
++ export MARIADBCLIENT_DIR=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3
++ MARIADBCLIENT_DIR=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3
++ export PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/bin:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/bin:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/bin:/astro/ferguson1/jlong/lsststack/Linux64/doxygen/1.8.5.lsst1/bin:/astro/ferguson1/jlong/lsststack/eups/bin:/astro/ferguson1/jlong/lsststack/Linux64/lsst/2016_01.0-8-g0fc2317+2/bin:/astro/ferguson1/jlong/lsststack/Linux64/scons/2.3.5/bin:/astro/ferguson1/jlong/lsststack/Linux64/sconsUtils/2016_01.0-4-g49bc714/bin:/astro/ferguson1/jlong/lsststack/Linux64/miniconda2/3.19.0.lsst4/bin:/user/jlong/miniconda3/bin:/user/jlong/.linuxbrew/bin:/opt/rh/devtoolset-3/root/usr/bin:/opt/rh/rh-ruby22/root/usr/bin:/opt/rh/git19/root/usr/bin:/home/jlong/bin:/usr/local/bin:/opt/rh/rh-java-common/root/usr/bin:/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/grp/software/Linux/Wolfram/Mathematica/9.0/bin:/grp/software/Linux/RH6/x86_64/sm-2.4.36/bin
++ PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/bin:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/bin:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/bin:/astro/ferguson1/jlong/lsststack/Linux64/doxygen/1.8.5.lsst1/bin:/astro/ferguson1/jlong/lsststack/eups/bin:/astro/ferguson1/jlong/lsststack/Linux64/lsst/2016_01.0-8-g0fc2317+2/bin:/astro/ferguson1/jlong/lsststack/Linux64/scons/2.3.5/bin:/astro/ferguson1/jlong/lsststack/Linux64/sconsUtils/2016_01.0-4-g49bc714/bin:/astro/ferguson1/jlong/lsststack/Linux64/miniconda2/3.19.0.lsst4/bin:/user/jlong/miniconda3/bin:/user/jlong/.linuxbrew/bin:/opt/rh/devtoolset-3/root/usr/bin:/opt/rh/rh-ruby22/root/usr/bin:/opt/rh/git19/root/usr/bin:/home/jlong/bin:/usr/local/bin:/opt/rh/rh-java-common/root/usr/bin:/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/grp/software/Linux/Wolfram/Mathematica/9.0/bin:/grp/software/Linux/RH6/x86_64/sm-2.4.36/bin
++ export LSST_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib
++ LSST_LIBRARY_PATH=/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/mariadbclient-10.1.11.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/gsl/1.16.lsst3/lib:/astro/ferguson1/jlong/lsststack/Linux64/fftw/3.3.4.lsst2/lib:/astro/ferguson1/jlong/lsststack/Linux64/cfitsio/3360.lsst4/lib
+ ./ups/eupspkg FLAVOR=Linux64 VERBOSE=0 config
cmake: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
+ exit -3


***** error: from /astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/build.log:
eups distrib: Failed to build mariadbclient-10.1.11.lsst3.eupspkg: Command:
	source /astro/ferguson1/jlong/lsststack/eups/bin/setups.sh; export EUPS_PATH=/astro/ferguson1/jlong/lsststack; (/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/build.sh) >> /astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/build.log 2>&1 4>/astro/ferguson1/jlong/lsststack/EupsBuildDir/Linux64/mariadbclient-10.1.11.lsst3/build.msg 
exited with code 253

Now, sure enough, the first place I thought to look (/usr/lib/) I see libstdc++.so.6 is missing. My question is, does that fully explain the problem? (In which case, I will ask our IT people.) Or could it be that this is an interaction between the Software-Collections-provided g++ and the build script?

josePhoenix · July 25, 2016, 8:56pm

A quick update thanks to a more savvy colleague: the devtoolset-3 Software Collection provides these library files for libstdc++. It’s not clear that the build script knows to look for them there, however.

/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2/32/libstdc++.so
/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2/libstdc++.a
/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2/libstdc++.so
/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2/libstdc++_nonshared.a

price · July 25, 2016, 9:20pm

One way for your compiler not to see those files is if you’re using a different compiler than the one installed in devtoolset-3. Can you ask your system which g++?

josePhoenix · July 25, 2016, 9:45pm

Sure, here you go:

17:44:11 science6:~ jlong$ which g++
/opt/rh/devtoolset-3/root/usr/bin/g++
17:44:13 science6:~ jlong$ g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.2/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/opt/rh/devtoolset-3/root/usr --mandir=/opt/rh/devtoolset-3/root/usr/share/man --infodir=/opt/rh/devtoolset-3/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --enable-languages=c,c++,fortran,lto --enable-plugin --with-linker-hash-style=gnu --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.9.2-20150212/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.9.2-20150212/obj-x86_64-redhat-linux/cloog-install --with-mpc=/builddir/build/BUILD/gcc-4.9.2-20150212/obj-x86_64-redhat-linux/mpc-install --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.9.2 20150212 (Red Hat 4.9.2-6) (GCC)

swinbank · July 26, 2016, 1:46pm

My guess is that this error isn’t coming from g++ but from CMake itself, which is not included in devtoolset-3: it should already be available on the system and linked against the system default libstdc++.

I see the following on a test machine:

$ lsb_release -d
Description:	CentOS release 6.8 (Final)

$ scl enable devtoolset-3 "which cmake"
/usr/bin/cmake

$ scl enable devtoolset-3 "ldd $(which cmake) | grep libstdc++"
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000036fe600000)

In other words, CMake isn’t coming out of some special devtoolset location, and it’s linked against /usr/lib64/libstdc++.so.6 (which exists!). Can try the same on your system?

swinbank · July 26, 2016, 1:48pm

(A simpler test would be just to try executing cmake directly and see if you get a similar error!)

josePhoenix · July 26, 2016, 6:13pm

Thanks John! cmake itself is indeed busted on this server. I’m going to file a ticket with our IT folks.