Build Issue when building from newstall.sh et al

Support
#1

I followed the directions at https://pipelines.lsst.io/install/newinstall.html
to build LSST.

I am running Ubuntu on an old laptop.

I ran the command ‘eups distrib install -t v12_0 lsst_apps’

It executed, without error, until it reached the step 40/62 afw 12.0.

48 hours later the following was writing to console. I ran build with -vvv to obtain logs. The build.log from lsst_stack/EupsBuildDir/Linux64/afw-12.0 is attached.

Keith Smith

  [ 40/62 ]  afw 12.0 ... Config parameter, MANIFEST_TAGGED_URL, not set; skipping
Config parameter, MANIFEST_FLAVOR_URL, not set; skipping
Trying retrieve using MANIFEST_URL to /tmp/eups/distrib/afw_26j1xoem
Looking on server for http://sw.lsstcorp.org/eupspkg/manifests/afw-12.0.manifest
Looking for mapping for cfitsio 3360.lsst4 Linux64
Looking for mapping for doxygen 1.8.5.lsst1 Linux64
Looking for mapping for eigen 3.2.5.lsst1 Linux64
Looking for mapping for fftw 3.3.4.lsst2 Linux64
Looking for mapping for gsl 1.16.lsst3 Linux64
Looking for mapping for mariadbclient 10.1.11.lsst3 Linux64
Looking for mapping for minuit2 5.34.14 Linux64
Looking for mapping for python 0.0.4 Linux64
Looking for mapping for python_d2to1 0.2.12.lsst1 Linux64
Looking for mapping for swig 3.0.2.lsst1 Linux64
Looking for mapping for astropy 0.0.1.lsst2 Linux64
Looking for mapping for boost 1.60+1 Linux64
Looking for mapping for numpy 0.0.2+1 Linux64
Looking for mapping for python_psutil 4.1.0+1 Linux64
Looking for mapping for pyyaml 3.11.lsst1+1 Linux64
Looking for mapping for scons 2.5.0.lsst1 Linux64
Looking for mapping for stsci_distutils 0.3.7.lsst1 Linux64
Looking for mapping for wcslib 5.13.lsst1 Linux64
Looking for mapping for matplotlib 0.0.2+1 Linux64
Looking for mapping for pyfits 3.4.0+4 Linux64
Looking for mapping for sconsUtils 12.0 Linux64
Looking for mapping for base 12.0 Linux64
Looking for mapping for ndarray 12.0 Linux64
Looking for mapping for pex_exceptions 12.0 Linux64
Looking for mapping for utils 12.0 Linux64
Looking for mapping for daf_base 12.0 Linux64
Looking for mapping for pex_logging 12.0 Linux64
Looking for mapping for pex_policy 12.0 Linux64
Looking for mapping for daf_persistence 12.0 Linux64
Looking for mapping for pex_config 12.0 Linux64
Looking for mapping for afw 12.0 Linux64
Using Distrib type: eupspkg
[dl] Config parameter, EUPSPKG_TAGGED_URL, not set; skipping
Config parameter, EUPSPKG_FLAVOR_URL, not set; skipping
Trying retrieve using EUPSPKG_URL to /tmp/eups/distrib/afw_si9k_dfz
Looking on server for http://sw.lsstcorp.org/eupspkg/products/afw-12.0.eupspkg
Building package: afw-12.0.eupspkg
Building in directory: /media/ksmith/MyPassport_2/lsst_stack/EupsBuildDir/Linux64/afw-12.0
Writing log to: /media/ksmith/MyPassport_2/lsst_stack/EupsBuildDir/Linux64/afw-12.0/build.log
[build] 

***** error: from /media/ksmith/MyPassport_2/lsst_stack/EupsBuildDir/Linux64/afw-12.0/build.log:
python/lsst/afw/math/statistics.i:42: Warning 504: Function lsst::afw::math::Statistics::Statistics(lsst::afw::image::Image< int > const &,lsst::afw::image::Mask< unsigned short > const &,lsst::afw::image::Image< float > const &,int const,lsst::afw::math::StatisticsControl const &) must have a return type. Ignored.
python/lsst/afw/math/statistics.i:42: Warning 504: Function lsst::afw::math::Statistics::Statistics(lsst::afw::image::Image< int > const &,lsst::afw::image::Mask< unsigned short > const &,lsst::afw::image::Image< float > const &,int const) must have a return type. Ignored.
python/lsst/afw/math/statistics.i:46: Warning 504: Function lsst::afw::math::Statistics::Statistics(lsst::afw::image::Mask< unsigned short > const &,lsst::afw::image::Mask< unsigned short > const &,lsst::afw::image::Mask< unsigned short > const &,int const,lsst::afw::math::StatisticsControl const &) must have a return type. Ignored.
python/lsst/afw/math/statistics.i:46: Warning 504: Function lsst::afw::math::Statistics::Statistics(lsst::afw::image::Mask< unsigned short > const &,lsst::afw::image::Mask< unsigned short > const &,lsst::afw::image::Mask< unsigned short > const &,int const) must have a return type. Ignored.
python/lsst/afw/math/background.i:98: Warning 504: Function lsst::afw::math::BackgroundMI::BackgroundMI(lsst::afw::image::Image< float > const &,lsst::afw::math::BackgroundControl const &) must have a return type. Ignored.
c++ -o python/lsst/afw/math/mathLib_wrap.os -c -std=c++11 -g -DLSST_LITTLE_ENDIAN=1 -O3 -fPIC -Iinclude -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/ndarray/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/pex_config/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/daf_persistence/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/pex_policy/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/pex_logging/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/daf_base/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/utils/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/pex_exceptions/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/base/12.0/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/wcslib/5.13.lsst1/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/cfitsio/3360.lsst4/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/gsl/1.16.lsst3/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/minuit2/5.34.14/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/fftw/3.3.4.lsst2/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/eigen/3.2.5.lsst1/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/mariadbclient/10.1.11.lsst3/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/boost/1.60+1/include -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/miniconda2/3.19.0.lsst4/include/python2.7 -I/media/ksmith/MyPassport_2/lsst_stack/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/site-packages/numpy/core/include python/lsst/afw/math/mathLib_wrap.cc
In file included from /media/ksmith/MyPassport_2/lsst_stack/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1781:0,
                 from /media/ksmith/MyPassport_2/lsst_stack/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /media/ksmith/MyPassport_2/lsst_stack/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from python/lsst/afw/math/mathLib_wrap.cc:4567:
/media/ksmith/MyPassport_2/lsst_stack/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.
scons: *** [python/lsst/afw/image/imageLib_wrap.os] Error 4
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build afw-12.0.eupspkg: Command:
    source /media/ksmith/MyPassport_2/lsst_stack/eups/bin/setups.sh; export EUPS_PATH=/media/ksmith/MyPassport_2/lsst_stack; (/media/ksmith/MyPassport_2/lsst_stack/EupsBuildDir/Linux64/afw-12.0/build.sh) >> /media/ksmith/MyPassport_2/lsst_stack/EupsBuildDir/Linux64/afw-12.0/build.log 2>&1 4>/media/ksmith/MyPassport_2/lsst_stack/EupsBuildDir/Linux64/afw-12.0/build.msg 
exited with code 252
Removing lockfile /media/ksmith/MyPassport_2/lsst_stack/.lockDir/exclusive-ksmith.19602

build.log (686.9 KB)

Stack installation failed on afw-13.0 -- OOM
#2

Ouch. What version of gcc is this?

It probably won’t help but v12_1 was released last week.

Build on Scientific Linux 7 fails for afw
#3

Hi Keith,

How much memory do you have on your laptop? Building afw is memory intensive and there’s definitely issues when you have less than 4GB of memory available.

#4

I have 3.9GB. Too little?
gcc => 4.8.4

#5

Looks like the OOM killer. Compiling imageLib_wrap.cc takes a lot of memory.

If you’re desperate, you might get away with it if you don’t use too many cores at once and have some swap available. I also suggest turning off debug symbols (SCONSFLAGS='debug=0') which will help save some memory usage.

It looks like you’re doing this on an external drive too. The compilation may take a loooong time…

You might consider trying to use a binary release on this machine.

#6

The external drive was to get some ‘breaks’ in the build to even look at the screen after screen saver timeouts. Also a place to put the build.log file and provide more ‘breaks’.
[This is not my first attempt at the build.]

Thanks, yes OOM seems to be the problem. I’ll consider another hardware platform. Go bigger or go home. :slight_smile:

I’ll make sure RHEL installs before I continue as I believe LSST has RHEL as its ‘preferred’ platform?

#7

Theoretically, yes. In practise, you’ll probably be able to find help if you have a problem on any *nix-like system. RHEL/CentOS, Ubuntu and OSX are, I think, the most common, but whatever you’re running, let us know if you have a problem and we’ll see what we can do.

#8

Have you tried installing the Anaconda package? That would get you a binary, without having to build anything. Instructions are here:

https://pipelines.lsst.io/install/conda.html

#9

Yes I was able to install the binary, ran the demo, and the compare was ‘ok’.

#10

I went into Github.com/lsst/afw and looked for imageLib_wrap.cc but I didn’t find the file there.

Is it somewhere else?

#11

It’s generated during the build process by swig.

#12

Thank you, said the blind man, who picked up his hammer and saw.

#13

Reporting another build data point.

I got some bigger iron. 12GB RAM 4 core i7 (1.75Ghz, 8 hyperthreads).

Performed the build to an external USB drive.

Successful build, time: 120minutes+/-5min

Maximum memory usage 14GB (2GB of swap created)

Many times, especially with AFW, all 8 hyperthreads running at 100% for an extended period of time.

Running Fedora v24. Ran ‘dnf upgrade’ before installing lsst pre-requisites, newstall, and ‘eups distrib install’

gcc 6.2.1

HYFTU

#14

Following up on this: I am trying to help someone build the stack on a machine with exactly 4GB of RAM. She is also getting a c++: internal compiler error: Killed (program cc1plus) error on afw. Is 4GB no longer enough to build the stack? Are there any clever workarounds I can pass along to try to get this through, or is 8GB the new requirement for building the stack?

Here are the last lines of her build.log

c++ -o python/lsst/afw/detection/_detectionLib.so -Wl,-rpath-link -Wl,/LSST/EupsBuildDir/Linux64/afw-13.0+2/afw-13.0+2/lib:/LSST/Linux64/pex_config/13.0+2/lib:/LSST/Linux64/daf_persistence/13.0+1/lib:/LSST/Linux64/pex_policy/13.0+2/lib:/LSST/Linux64/daf_base/13.0+2/lib:/LSST/Linux64/utils/13.0+2/lib:/LSST/Linux64/pex_exceptions/13.0+1/lib:/LSST/Linux64/log/13.0+2/lib:/LSST/Linux64/healpy/1.8.1.lsst2+5/lib:/LSST/Linux64/galsim/1.4.3/lib:/LSST/Linux64/base/13.0+1/lib:/LSST/Linux64/tmv/0.73+5/lib:/LSST/Linux64/sncosmo/12.0-1-g5e70e90+2/lib:/LSST/Linux64/oorb/lsst-dev-g86ef29d214/lib:/LSST/Linux64/log4cxx/0.10.0.lsst7/lib:/LSST/Linux64/astrometry_net/0.67.123ff3e.lsst1+1/lib:/LSST/Linux64/wcslib/5.13.lsst1/lib:/LSST/Linux64/boost/1.60.lsst1+1/lib:/LSST/Linux64/apr_util/1.5.4/lib:/LSST/Linux64/minuit2/5.34.14/lib:/LSST/Linux64/mariadbclient/10.1.21.lsst1/lib:/LSST/Linux64/gsl/1.16.lsst3/lib:/LSST/Linux64/freetds/0.91.112.lsst2/lib:/LSST/Linux64/fftw/3.3.4.lsst2/lib:/LSST/Linux64/cfitsio/3360.lsst4/lib:/LSST/Linux64/apr/1.5.2/lib -shared python/lsst/afw/detection/detectionLib_wrap.os -Llib -L/LSST/Linux64/wcslib/5.13.lsst1/lib -L/LSST/Linux64/cfitsio/3360.lsst4/lib -L/LSST/Linux64/gsl/1.16.lsst3/lib -L/LSST/Linux64/minuit2/5.34.14/lib -L/LSST/Linux64/fftw/3.3.4.lsst2/lib -L/LSST/Linux64/daf_persistence/13.0+1/lib -L/LSST/Linux64/mariadbclient/10.1.21.lsst1/lib -L/LSST/Linux64/pex_policy/13.0+2/lib -L/LSST/Linux64/log/13.0+2/lib -L/LSST/Linux64/log4cxx/0.10.0.lsst7/lib -L/LSST/Linux64/daf_base/13.0+2/lib -L/LSST/Linux64/utils/13.0+2/lib -L/LSST/Linux64/pex_exceptions/13.0+1/lib -L/LSST/Linux64/base/13.0+1/lib -L/LSST/Linux64/boost/1.60.lsst1+1/lib -L/LSST/Linux64/miniconda2/4.2.12.lsst1/lib/python2.7/config -lafw -lwcs -lcfitsio -lgsl -lgslcblas -lMinuit2 -lfftw3f -lfftw3 -ldaf_persistence -lboost_serialization -lmysqlclient_r -lpex_policy -lboost_filesystem -lboost_system -llog -llog4cxx -ldaf_base -lutils -lpex_exceptions -lbase -lboost_regex -lpthread -lutil -ldl -lpython2.7
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
scons: *** [python/lsst/afw/image/imageLib_wrap.os] Error 4
/LSST/Linux64/miniconda2/4.2.12.lsst1/lib/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tmpnam':
/home/ilan/minonda/conda-bld/work/Python-2.7.12/./Modules/posixmodule.c:7631: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
/LSST/Linux64/miniconda2/4.2.12.lsst1/lib/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tempnam':
/home/ilan/minonda/conda-bld/work/Python-2.7.12/./Modules/posixmodule.c:7578: warning: the use of `tempnam' is dangerous, better use `mkstemp'
scons: building terminated because of errors.
+ exit -4
#15

You could try moving to a newer version of the stack - that failure seems to have happened when building a Swig-generated file, and it’d be reasonable to hope (we haven’t tested, AFAIK) that the memory requirements on pybind11 builds are lower.

#16

Use a single core and turn off debug symbols. You need to set EUPSPKG_SCONSFLAGS or SCONSFLAGS (according to how you’re building, using eups or scons) to include -j 1 debug=0.

Stack installation failed on afw-13.0 -- OOM
#17

Thanks, Jim. Building a pbyind11 version worked.

#18

Paul’s fix also seems to have worked.