Afw install failed

Support
#1

Dear all,
I am trying to install lsst-stack on Ubuntu 16.10.
Using "Source Installation with newinstall.sh"
It works well but when I step into “Install packages” I meet error from afw:

$ eups distrib install -t v12_0 lsst_apps
$

but …

   [  1/62 ]  cfitsio 3360.lsst4 (already installed)                     done. 
   [  2/62 ]  doxygen 1.8.5.lsst1 (already installed)                    done. 
   [  3/62 ]  eigen 3.2.5.lsst1 (already installed)                      done. 
   [  4/62 ]  fftw 3.3.4.lsst2 (already installed)                       done. 
   [  5/62 ]  gsl 1.16.lsst3 (already installed)                         done. 
   [  6/62 ]  mariadbclient 10.1.11.lsst3 (already installed)            done. 
   [  7/62 ]  minuit2 5.34.14 (already installed)                        done. 
   [  8/62 ]  python 0.0.4 (already installed)                           done. 
   [  9/62 ]  python_d2to1 0.2.12.lsst1 (already installed)              done. 
   [ 10/62 ]  swig 3.0.2.lsst1 (already installed)                       done. 
   [ 11/62 ]  xpa 2.1.15.lsst3 (already installed)                       done. 
   [ 12/62 ]  astropy 0.0.1.lsst2 (already installed)                    done. 
   [ 13/62 ]  boost 1.60+1 (already installed)                           done. 
   [ 14/62 ]  mysqlpython 1.2.3.lsst2+2 (already installed)              done. 
   [ 15/62 ]  numpy 0.0.2+1 (already installed)                          done. 
   [ 16/62 ]  python_psutil 4.1.0+1 (already installed)                  done. 
   [ 17/62 ]  pyyaml 3.11.lsst1+1 (already installed)                    done. 
   [ 18/62 ]  scons 2.5.0.lsst1 (already installed)                      done. 
   [ 19/62 ]  stsci_distutils 0.3.7.lsst1 (already installed)            done. 
   [ 20/62 ]  wcslib 5.13.lsst1 (already installed)                      done. 
   [ 21/62 ]  astrometry_net 0.50.lsst3+1 (already installed)            done. 
   [ 22/62 ]  esutil 0.5.3+1 (already installed)                         done. 
   [ 23/62 ]  matplotlib 0.0.2+1 (already installed)                     done. 
   [ 24/62 ]  pyfits 3.4.0+4 (already installed)                         done. 
   [ 25/62 ]  scipy 0.0.1.lsst1+1 (already installed)                    done. 
   [ 26/62 ]  sconsUtils 12.0 (already installed)                        done. 
   [ 27/62 ]  astrometry_net_data 10.0+77 (already installed)            done. 
   [ 28/62 ]  base 12.0 (already installed)                              done. 
   [ 29/62 ]  geom 12.0 (already installed)                              done. 
   [ 30/62 ]  lmfit 0.9.3+2 (already installed)                          done. 
   [ 31/62 ]  psfex 12.0 (already installed)                             done. 
   [ 32/62 ]  ndarray 12.0 (already installed)                           done. 
   [ 33/62 ]  pex_exceptions 12.0 (already installed)                    done. 
   [ 34/62 ]  utils 12.0 (already installed)                             done. 
   [ 35/62 ]  daf_base 12.0 (already installed)                          done. 
   [ 36/62 ]  pex_logging 12.0 (already installed)                       done. 
   [ 37/62 ]  pex_policy 12.0 (already installed)                        done. 
   [ 38/62 ]  daf_persistence 12.0 (already installed)                   done. 
   [ 39/62 ]  pex_config 12.0 (already installed)                        done. 
  [ 40/62 ]  afw 12.0 ... 




***** error: from /home/yuanshuo/Works_Neptune/LSST/EupsBuildDir/Linux64/afw-12.0/build.log:
  File "/home/yuanshuo/Works_Neptune/LSST/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/site-packages/matplotlib/cbook.py", line 1186, in listFiles
    for dirname, dirs, files in os.walk(root):
  File "/home/yuanshuo/Works_Neptune/LSST/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/yuanshuo/Works_Neptune/LSST/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/os.py", line 286, in walk
    if isdir(join(top, name)):
  File "/home/yuanshuo/Works_Neptune/LSST/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

----------------------------------------------------------------------
Ran 17 tests in 0.710s

FAILED (errors=3, skipped=3)

I do not understand why the "decode byte 0xe4 " here.

Could you help me on this? Thanks!

Shuo

#2

Can you please try the newer v12_1 release? Secondly, do you have a unicode file system?

#3

Can you please also give more of the stack trace from that file? It would help to know which test file was failing.

#4

That is seemingly an “a" with umlaut: ä in unicode.

#5

Thank you!

The file build.log is too big…

But I find

$ locale
LANG=zh_CN.UTF-8
LANGUAGE=zh_CN:en_AU:en_CA:en_GB:en_NZ:en_US:en
LC_CTYPE="zh_CN.UTF-8"
LC_NUMERIC=zh_CN.UTF-8
LC_TIME=zh_CN.UTF-8
LC_COLLATE="zh_CN.UTF-8"
LC_MONETARY=zh_CN.UTF-8
LC_MESSAGES="zh_CN.UTF-8"
LC_PAPER=zh_CN.UTF-8
LC_NAME=zh_CN.UTF-8
LC_ADDRESS=zh_CN.UTF-8
LC_TELEPHONE=zh_CN.UTF-8
LC_MEASUREMENT=zh_CN.UTF-8
LC_IDENTIFICATION=zh_CN.UTF-8
LC_ALL=
#6

The bit we needed was the test failure reports right at the end. The default dump in the eups error report is just the final few lines and truncates a bit too early.

That might be the problem. We’ve only really tested with the C locale. Hopefully Python 3 will make a lot of this easier (v12.1 does have experimental Python 3 support for the bulk of lsst_apps and v13.0 will have experimental support for all of lsst_apps).

#7

The log file is here build.log.cut (10.8 KB)

Thanks!

#8

Thank you:

Traceback (most recent call last):
  File "tests/rgb.py", line 335, in testWriteStarsLegacyAPI
    self.writeFileLegacyAPI(fileName)
  File "tests/rgb.py", line 330, in writeFileLegacyAPI
    rgbImage.write(fileName)
  File "/home/yuanshuo/Works_Neptune/LSST/EupsBuildDir/Linux64/afw-12.0/afw-12.0/python/lsst/afw/display/rgb.py", line 401, in write
    writeRGB(fileName, self.rgb)
  File "/home/yuanshuo/Works_Neptune/LSST/EupsBuildDir/Linux64/afw-12.0/afw-12.0/python/lsst/afw/display/rgb.py", line 379, in writeRGB
    matplotlib.image.imsave(fileName, rgbImage)

So all the failures relate to using matplotlib to save an RGB image to disk. The trigger is because it’s rebuilding the font cache (this gets disabled in our tests because it triggered race conditions on older matplotlib versions [pre v1.5.1]) and somehow the environment is messed up.

Can you try forcing your locale to C and seeing if that gets past the problem?

#9

I try to set the C locale by export LC_ALL=C

$ locale
LANG=en_US.UTF-8
LANGUAGE=zh_CN:en_AU:en_CA:en_GB:en_NZ:en_US:en
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

However, it still doesn’t help:

$eups distrib install -t v12_0 lsst_apps

   [ 37/62 ]  pex_policy 12.0 (already installed)                        done. 
   [ 38/62 ]  daf_persistence 12.0 (already installed)                   done. 
   [ 39/62 ]  pex_config 12.0 (already installed)                        done. 
  [ 40/62 ]  afw 12.0 ... 

***** error: from /home/yuanshuo/Works_Neptune/LSST/EupsBuildDir/Linux64/afw-12.0/build.log:
  File "/home/yuanshuo/Works_Neptune/LSST/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/site-packages/matplotlib/cbook.py", line 1186, in listFiles
    for dirname, dirs, files in os.walk(root):
  File "/home/yuanshuo/Works_Neptune/LSST/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/home/yuanshuo/Works_Neptune/LSST/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/os.py", line 286, in walk
    if isdir(join(top, name)):
  File "/home/yuanshuo/Works_Neptune/LSST/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

I looked at the log file, it seems a little bit different build_c_locale.log.cut (10.5 KB)

#10

I am going to guess that matplotlib does not work at all in the LSST conda environment. Do you have any matplotlib code you can test with that python? I’m not entirely sure what directory matplotlib is scanning but it seems to be finding a file with an “a umlaut” in its name.