I installed the hscPipe 7.9.1 with binary distribution (by setup EUPS_PKGROOT ) at Fedora30. It start from scratch. Although I got one library issue and two warnings (see below), the installation seems successful. However, I got library error when I run some test scripts (ingestCalibs.py, hscIngestImages.py, singleFrameDriver.py). I wonder anyone can give a advise to fix it?
Charles
Main error message:
(lsst-scipipe-1172c30) [ytchen@ichiro hscPipe7.9.1]$ ingestCalibs.py
Traceback (most recent call last):
File “/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.9.1-hsc/bin/ingestCalibs.py”, line 2, in
from lsst.pipe.tasks.ingestCalibs import IngestCalibsTask
File “/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/7.9.1-hsc/python/lsst/pipe/tasks/ingestCalibs.py”, line 5, in
from lsst.afw.fits import readMetadata
File “/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/fits/init.py”, line 1, in
from .fitsLib import *
File “/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/afw/7.6-hsc/python/lsst/afw/fits/fitsLib.py”, line 1, in
from .fits import *
ImportError: /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/daf_base/7.0-hsc/lib/libdaf_base.so: undefined symbol: _ZN5boost16re_detail_10690012perl_matcherIN9__gnu_cxx17__normal_iteratorIPKcSsEESaINS_9sub_matchIS6_EEENS_12regex_traitsIcNS_16cpp_regex_traitsIcEEEEE14construct_initERKNS_11basic_regexIcSD_EENS_15regex_constants12_match_flagsE
Issue 1:
I fixed it by replacing /array/users/ytchen/hscPipe/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/../lib/libstdc++.so.6 with /usr/lib64/libstdc++.so.6 (after running newinstall.sh).
ImportError while importing test module ‘/array/users/ytchen/hscPipe/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/astshim-7.3-hsc/astshim-7.3-hsc/tests/test_specFrame.py’.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_specFrame.py:4: in <module>
import astshim as ast
python/astshim/init.py:26: in <module>
from .base import *
E ImportError: /array/users/ytchen/hscPipe/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/…/lib/libstdc++.so.6: version `GLIBCXX_3.4.26’ not found (required by /array/users/ytchen/hscPipe/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/astshim-7.3-hsc/astshim-7.3-hsc/lib/libastshim.so)
Two warnings when I install hscPipe 7.9.1:
[ 85/97 ] meas_deblender 7.0-hsc+1 (Linux64) … Warning: trouble retrieving meas_deblender-7.0-hsc+1@Linux64.tar.gz: IncompleteRead(1573727 bytes read, 8955030 more expected)
[ 89/97 ] meas_modelfit 7.0-hsc+1 (Linux64) done.
Failed to contact URL http://tigress-web.princeton.edu/~HSC/hscPipe7/Linux64/ ([Errno -2] Name or service not known)
Actually, I encounter a error when I am installing from source code. It look likes a threading issue, but I couldn’t figure it out …
Here is the error message:
[ 55/97 ] base 7.0-hsc …
***** error: from /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.log:
warn(ConfigurationMissingWarning(msg))
tests/test_packages.py::PackagesTestCase::testPackages
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/python_future/0.16.0+6/lib/python/future-0.16.0-py3.7.egg/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from ‘collections’ instead of from ‘collections.abc’ is deprecated, and in 3.8 it will stop working
from collections import Iterable
/array/users/ytchen/hscPipe7.9.1/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/astropy/config/configuration.py:536: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
warn(ConfigurationMissingWarning(msg))
– Docs: http://doc.pytest.org/en/latest/warnings.html
========= 1 failed, 14 passed, 1 skipped, 4 warnings in 70.02 seconds ==========
Coverage.py warning: No data was collected. (no-data-collected)
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml.failed
The following tests failed:
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build base-7.0-hsc.eupspkg: Command:
source “/array/users/ytchen/hscPipe7.9.1/eups/2.1.5/bin/setups.sh”; export EUPS_PATH="/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30"; (/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.sh) >> /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.log 2>&1 4>/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.msg
exited with code 252
This is the last part of build.log
============================= test session starts ==============================
platform linux – Python 3.7.2, pytest-3.6.2, py-1.7.0, pluggy-0.6.0
rootdir: /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc, inifile: setup.cfg
plugins: flake8-1.0.4, xdist-1.20.1, forked-0.2, session2file-0.1.9, cov-2.5.1, remotedata-0.3.1, openfiles-0.3.2, doctestplus-0.3.0, arraydiff-0.3
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I
passed
gw0 [16] / gw1 [16] / gw2 [16] / gw3 [16] / gw4 [16] / gw5 [16] / gw6 [16] / gw7 [16] / gw8 [16] / gw9 [16] / gw10 [16] / gw11 [16] / gw12 [16] / gw13 [16] / gw14 [16] / gw15 [16] / gw16 [16] / gw17 [16] / gw18 [16] / gw19 [16] / gw20 [16] / gw21 [16] / gw22 [16] / gw23 [16] / gw24 [16] / gw25 [16] / gw26 [16] / gw27 [16] / gw28 [16] / gw29 [16] / gw30 [16] / gw31 [16]
scheduling tests via LoadScheduling
s.[gw15] node down: Not properly terminated
fSlave restarting disabled
…
=================================== FAILURES ===================================
____________________________ tests/test_threads.py _____________________________
[gw15] linux – Python 3.7.2 /array/users/ytchen/hscPipe7.9.1/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/python3.7
Slave ‘gw15’ crashed while running ‘tests/test_threads.py::ThreadsTestCase::testApi’
generated xml file: /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml
----------- coverage: platform linux, python 3.7.2-final-0 -----------
Name Stmts Miss Branch BrPart Cover
----------------------------------------------------------------
python/lsst/init.py 2 0 0 0 100%
python/lsst/base/init.py 4 0 0 0 100%
python/lsst/base/packages.py 118 23 40 2 80%
python/lsst/base/version.py 5 0 0 0 100%
python/lsstcppimport.py 1 0 0 0 100%
python/lsstimport.py 50 20 19 7 54%
tests/test_packages.py 50 1 2 1 96%
tests/test_threads.py 12 8 2 1 36%
----------------------------------------------------------------
TOTAL 242 52 63 11 76%
Coverage HTML written to dir tests/.tests/pytest-base.xml-htmlcov
Coverage XML written to file tests/.tests/pytest-base.xml-cov-base.xml
---------------------- coverage: failed slaves -----------------------
The following slaves failed to return coverage data, ensure that pytest-cov is installed on these slaves.
gw15
=============================== warnings summary ===============================
tests/test_packages.py::PackagesTestCase::testPython
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/python_future/0.16.0+6/lib/python/future-0.16.0-py3.7.egg/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from ‘collections’ instead of from ‘collections.abc’ is deprecated, and in 3.8 it will stop working
from collections import Iterable
/array/users/ytchen/hscPipe7.9.1/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/astropy/config/configuration.py:536: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
warn(ConfigurationMissingWarning(msg))
tests/test_packages.py::PackagesTestCase::testPackages
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/Linux64/python_future/0.16.0+6/lib/python/future-0.16.0-py3.7.egg/past/types/oldstr.py:5: DeprecationWarning: Using or importing the ABCs from ‘collections’ instead of from ‘collections.abc’ is deprecated, and in 3.8 it will stop working
from collections import Iterable
/array/users/ytchen/hscPipe7.9.1/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/astropy/config/configuration.py:536: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
warn(ConfigurationMissingWarning(msg))
– Docs: http://doc.pytest.org/en/latest/warnings.html
========= 1 failed, 14 passed, 1 skipped, 4 warnings in 70.02 seconds ==========
Coverage.py warning: No data was collected. (no-data-collected)
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml.failed
The following tests failed:
/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/.tests/pytest-base.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
I replace libstdc++.so.6 in miniconda with that in system, because I encountered following error message (only post part of long message). Actually it really confuses me, the error message tell that the miniconda don’t include GLIBCXX_3.4.26, which is the request of eups package…
ImportError while importing test module ‘/array/users/ytchen/hscPipe/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/astshim-7.3-hsc/astshim-7.3-hsc/tests/test_specFrame.py’.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_specFrame.py:4: in <module>
import astshim as ast
python/astshim/init.py:26: in <module>
from .base import *
E ImportError: /array/users/ytchen/hscPipe/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/bin/…/lib/libstdc++.so.6: version `GLIBCXX_3.4.26’ not found (required by /array/users/ytchen/hscPipe/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/astshim-7.3-hsc/astshim-7.3-hsc/lib/libastshim.so)
Should I use older version? any suggestion for version?
I tried following case, all encountered the same issue:
1 core without EUPSPKG_NJOBS=1
1 core with EUPSPKG_NJOBS=1
16 core without EUPSPKG_NJOBS=1
16 core with EUPSPKG_NJOBS=1
I still saw 32 running jobs of python3.7 (via top) when the installation of base for four cases above.
I naively tried to use the binary installation for base, afw and daf_butler individually, which all get this kind of error. They can pass the installation but I never get the whole installation done. However, I guess that I will encounter ABI incompatibilities somewhere.
I start over the whole installation with EUPSPKG_NJOBS=1, then get this error message
[ 55/97 ] base 7.0-hsc …
***** error: from /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.log:
plugins: flake8-1.0.4, xdist-1.20.1, forked-0.2, session2file-0.1.9, cov-2.5.1, remotedata-0.3.1, openfiles-0.3.2, doctestplus-0.3.0, arraydiff-0.3
collected 16 items
run-last-failure: run all (no recorded failures)
doc/conf.py . [ 6%]
python/lsstDebug.py . [ 12%]
python/lsstcppimport.py . [ 18%]
python/lsstimport.py . [ 25%]
python/lsst/init.py . [ 31%]
python/lsst/base/init.py . [ 37%]
python/lsst/base/packages.py . [ 43%]
python/lsst/base/version.py . [ 50%]
tests/testModuleImporter2.py . [ 56%]
tests/test_packages.py … [ 87%]
tests/test_threads.py .sh: line 1: 73556 Segmentation fault (core dumped) ( TRAVIS=1 pytest -Wd --lf --lfnf=all --junit-xml=tests/.tests/pytest-base.xml --session2file=tests/.tests/pytest-base.xml.out --junit-prefix=base --cov=. --cov-branch --cov-report=term --cov-report=xml:‘tests/.tests/pytest-base.xml-cov-base.xml’ --cov-report=html:‘tests/.tests/pytest-base.xml-htmlcov’ )
Global pytest run: failed with 139
mv: cannot stat ‘tests/.tests/pytest-base.xml.out’: No such file or directory
scons: *** [tests/.tests/pytest-base.xml] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build base-7.0-hsc.eupspkg: Command:
source “/array/users/ytchen/hscPipe7.9.1/eups/2.1.5/bin/setups.sh”; export EUPS_PATH="/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30"; (/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.sh) >> /array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.log 2>&1 4>/array/users/ytchen/hscPipe7.9.1/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/build.msg
exited with code 252
I notice that these is a note at $HSCPATH/stack/miniconda3-4.5.12-1172c30/EupsBuildDir/Linux64/base-7.0-hsc/base-7.0-hsc/tests/test_threads.py. The EUPSPKG_NJOBS=1 could be the cause of the error: tests/test_threads.py .sh: line 1: 73556 Segmentation fault (core dumped) But I still have not solution for solving the installation of base 7.0-hsc…
import unittest
class ThreadsTestCase(unittest.TestCase):
def testApi(self):
from lsst.base import haveThreads
self.assertIsInstance(haveThreads(), bool)
from lsst.base import setNumThreads # noqa F401
# Raises lsst::base::NoThreadsException which is untranslated
# when threading library is missing, this needs fixing!
# self.assertIs(setNumThreads(4), None)
from lsst.base import getNumThreads
self.assertIsInstance(getNumThreads(), int)
from lsst.base import disableImplicitThreading
self.assertIsInstance(disableImplicitThreading(), bool)
Then look up a compiler in the ABI table under point 4; I think that’s the latest version you can use, but I’m not sure.
For reference, on the LSST development machine, I get GLIBCXX_3.4.25 in the shared stack, and we regularly compile with gcc 6.3.1. The compiler version listed in the ABI table is gcc 8.0.0.
I think you are right about this. The new/older gcc probably causes unexpected issue, at least, I had confirmed older one has issues. Unfortunately, the OS of the purposed cluster I am trying to install hscPipe 7.9.1 is Fedora 30, I haven’t found a easy way to install devtoolset-6 yet, still working on that.
I get GLIBCXX_3.4.27 for gcc 9.2.1, so it’s too new… So this should be the cause of potential following unexpected issue, I guess.
#########################
In the mean time, I had successfully installed hscPipe 7.9.1 at two test PCs:
(1) CentOS 6 (default gcc is 4.4.7) with devtoolset-6
(2) whole new installation of CentOS 7 (default gcc is 4.8.5 ) with devtoolset-6
However, I found a potential issue/bug/prerequisite in (1) case. The old-installed fftw in (1) is version 3.2.1-3.1.el6, which produce the following failure error and crash entire installation during the installation of galsim 1.5.1.lsst1-11-gfba72ad:
/opt/rh/devtoolset-6/root/usr/libexec/gcc/x86_64-redhat-linux/6.3.1/ld: /usr/local/lib/libfftw3.a(hc2rIII_2.o): relocation R_X86_64_32 against `.rodata’ can not be used when making a shared object; recompile with -fPIC
Once I replace /usr/local/lib/libfftw3.a with the one (fftw 3.3.3-8.el7) from (2) , the whole installation is very smooth and successful. However, I am so curious about why eups installation didn’t use the one in miniconda (fftw 3.3.4.lsst2), or the ones in LD_LIBRARY_PATH or LIBRARY_PATH. The failure build log said “Found static library: /usr/local/lib/libfftw3.a” even I added the paths of new version or minconda fftw into LD_LIBRARY_PATH or LIBRARY_PATH. I thought the propose of miniconda is to provide a identical installation environment (except gcc), so why the installation didn’t use the one in miniconda?