Failure to build daf_butler

ktl · August 23, 2022, 12:32am

Sorry, it looks like there was a problem in our release process and the wrong version of newinstall.sh got tagged for 23.0.2. Please use the version for 23.0.1 instead in the meantime.

plah · August 24, 2022, 7:10am

Hello K-T Lim,
I tried the 23.0.1 release and it fails at the same point in the installation. I tried an earlier release (22.0.1) and the installation fails much earlier in the process.

timj · August 30, 2022, 3:41pm

Which conda environment did it install? (ie which version of rubin-env is it using?). The rubin-env for 23.0.1 installs should be using v0.8.1 which should pin astropy.

plah · August 30, 2022, 11:55pm

The code installs rubin-env=0.8.1.

timj · August 31, 2022, 2:35pm

Can you confirm your astropy version?

plah · August 31, 2022, 11:33pm

I am running astropy 5.1.

timj · September 1, 2022, 9:35pm

I am reminded that we don’t pin the astropy version in the rubin-env used by v23. We had back-ported the fix to support both astropy v4 and v5 but that is in the v23.0.3 release candidate that should be coming out at some point.

Is there a reason why you can’t use a newer release? v23 is derived from the codebase as it was back in October 2021. The v24.0 release is going to be based on the w_2022_28 tag so you should be safe installing that weekly and using it instead of v23.

plah · September 2, 2022, 1:46am

How do I go about installing the weekly release?

timj · September 2, 2022, 8:37pm

Where you currently ask for v23_0_2 from eups distrib install you instead ask for w_2022_28. You will need to have the 4.x rubin-env (which is what you will get from the main branch for newinstall.sh)

plah · September 5, 2022, 12:25am

I tried:

eups distrib install -t w_2022_28 lsst_distrib

It gets two steps further in the installation process before encountering an error.

The error mes

sage it outputs is:

[ 31/85 ] afw gba47b54d5d+94dc90c3ea …

***** error: from /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/build.log:
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/footprint1.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_imageHash.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_storable.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_key.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/pytest-afw.xml.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_simpleGenericMap.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/background.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_filterLabel.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/stacker.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/statistics.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/ticket1145.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_schema.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_functorKeys.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/image.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/rowColumnStats.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/ellipse.failed
36 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

exit -4
eups distrib: Failed to build afw-gba47b54d5d+94dc90c3ea.eupspkg: Command:
source “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/eups/bin/setups.sh”; export EUPS_PATH="/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0"; (/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/build.sh) >> /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/build.log 2>&1 4>/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/build.msg
exited with code 252

I have attached the log build.log file that goes with the error.
build.log (1.5 MB)

timj · September 5, 2022, 4:38pm

This is the error:

tests/maskedImage

tests/maskedImage: symbol lookup error: /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/lib/libafw.so: undefined symbol: _Py_Dealloc

It seems that a recent change to how linking worked in the conda infrastructure exposed a bug that we fixed on DM-35600. You will have to use at least w_2022_30 to get the build to go past that.

@yusra I think we will have to back-port that ticket to the v24 release.

plah · September 6, 2022, 12:05am

Hello Tim,

we seem to be slowly inching forward. I ran:

eups distrib install -t w_2022_30 lsst_distrib

and got to [ 46/84 ] before getting an error which halted installation. The exact error message is:

[ 46/84 ] meas_base gcf00a76f74+12dc3af6bc …

***** error: from /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/build.log:
Coverage XML written to file tests/.tests/pytest-meas_base.xml-cov-meas_base.xml

============================= slowest 5 durations ==============================
20.95s call tests/test_GaussianFlux.py::GaussianFluxTestCase::testMonteCarlo
2.41s call tests/test_SdssShape.py::SdssShapeTestCase::testMonteCarlo
2.15s call tests/test_ScaledApertureFlux.py::ScaledApertureFluxTestCase::testApertureTruncated
2.01s call tests/test_PsfFlux.py::PsfFluxTestCase::testMonteCarlo
1.89s call tests/test_SdssCentroid.py::SdssCentroidTestCase::testMonteCarlo
=========================== short test summary info ============================
FAILED tests/test_diaCalculationPlugins.py::TestSkewDiaPsFlux::testCalculate
================= 1 failed, 256 passed, 73 warnings in 30.70s ==================
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/meas_base-gcf00a76f74+12dc3af6bc/tests/.tests/pytest-meas_base.xml.failed
The following tests failed:
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/meas_base-gcf00a76f74+12dc3af6bc/tests/.tests/pytest-meas_base.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

exit -4
eups distrib: Failed to build meas_base-gcf00a76f74+12dc3af6bc.eupspkg: Command:
source “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/eups/bin/setups.sh”; export EUPS_PATH="/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0"; (/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/build.sh) >> /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/build.log 2>&1 4>/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/build.msg
exited with code 252

I have attached the log build.log file that goes with the error.
build.log (837.7 KB)

timj · September 6, 2022, 3:17pm

This is DM-35777. Rather than trying to pick the oldest weekly that might possibly work (to keep you as close to v23 as we can) you should pick something very new like w_2022_35 (or _36 if that’s out).

Thanks for reporting the v23 build problem. We will have to backport the butler test fix to 23.0.3 or pin astropy in that conda env.

plah · September 7, 2022, 4:49am

Hello Tim,

we are slowly getting there. I ran

eups distrib install -t w_2022_36 lsst_distrib

which should be the latest weekly. The installation got further stopping at:

[ 61/86 ] meas_extensions_shapeHSM g6072bfb783+8f013e7049

The error message this time was:

[ 61/86 ] meas_extensions_shapeHSM g6072bfb783+8f013e7049 …

***** error: from /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_extensions_shapeHSM-g6072bfb783+8f013e7049/build.log:
include/lsst/meas/extensions/shapeHSM/HsmAdapter.h:39:24: error: no matching function for call to ‘galsim::ImageView::ImageView(double*&, const std::shared_ptr&, int, const int&, const galsim::Bounds&)’
39 | return galsim::ImageView(ptr, _owner, 1, stride, bounds);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from src/HsmMoments.cc:32:
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:487:9: note: candidate: ‘galsim::ImageView::ImageView(galsim::ImageAlloc&) [with T = double]’
487 | ImageView(ImageAlloc& rhs) : BaseImage(rhs) {}
| ^~~~~~~~~
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:487:9: note: candidate expects 1 argument, 5 provided
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:479:9: note: candidate: ‘galsim::ImageView::ImageView(const galsim::ImageView&) [with T = double]’
479 | ImageView(const ImageView& rhs) : BaseImage(rhs) {}
| ^~~~~~~~~
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:479:9: note: candidate expects 1 argument, 5 provided
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:469:9: note: candidate: ‘galsim::ImageView::ImageView(T*, const T*, int, const std::shared_ptr<_Tp>&, int, int, const galsim::Bounds&) [with T = double]’
469 | ImageView(T* data, const T* maxptr, int nElements, const shared_ptr& owner,
| ^~~~~~~~~
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:469:9: note: candidate expects 7 arguments, 5 provided
scons: *** [src/HsmShape.os] Error 1
scons: *** [src/HsmMoments.os] Error 1
scons: building terminated because of errors.

exit -4
eups distrib: Failed to build meas_extensions_shapeHSM-g6072bfb783+8f013e7049.eupspkg: Command:
source “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/eups/bin/setups.sh”; export EUPS_PATH="/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0"; (/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_extensions_shapeHSM-g6072bfb783+8f013e7049/build.sh) >> /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_extensions_shapeHSM-g6072bfb783+8f013e7049/build.log 2>&1 4>/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_extensions_shapeHSM-g6072bfb783+8f013e7049/build.msg
exited with code 252

I have attached the log build.log file that goes with the error.
build.log (1.2 MB)

timj · September 7, 2022, 4:51am

I think you need to be using the 4.1.0 rubin-env that pins the galsim version – they changed the API.

plah · September 7, 2022, 5:36am

How do I go about doing that?

timj · September 8, 2022, 4:43am

Run newinstall.sh with the version that exists on main in the repo https://github.com/lsst/lsst repo.

plah · September 8, 2022, 6:59am

Hello Tim,

I tried using the different newinstall.sh. It set some environmental variables that the code complained about. However it still exited the installation at the same place with the same errors. Do you have anything else I can try?

timj · September 8, 2022, 3:43pm

What version of rubin-env do you have?

github.com

lsst/lsst/blob/main/scripts/newinstall.sh#L21

      
        
            
            
set -Eeo pipefail
            
            
LSST_EUPS_PKGROOT_BASE_URL=${LSST_EUPS_PKGROOT_BASE_URL:-https://eups.lsst.codes/stack}
            LSST_EUPS_USE_TARBALLS=${LSST_EUPS_USE_TARBALLS:-false}
            LSST_EUPS_USE_EUPSPKG=${LSST_EUPS_USE_EUPSPKG:-true}
            
            
LSST_MINICONDA_VERSION=${LSST_MINICONDA_VERSION:-py38_4.9.2}
            # this git ref controls which set of conda packages are used to initialize the
            # the default conda env defined in scipipe_conda_env git package (RFC-553).
            LSST_SPLENV_REF=${LSST_SPLENV_REF:-${LSST_LSSTSW_REF:-4.1.0}}
            LSST_MINICONDA_BASE_URL=${LSST_MINICONDA_BASE_URL:-https://repo.continuum.io/miniconda}
            LSST_CONDA_CHANNELS=${LSST_CONDA_CHANNELS:-"conda-forge"}
            LSST_CONDA_ENV_NAME=${LSST_CONDA_ENV_NAME:-lsst-scipipe-${LSST_SPLENV_REF}}
            LSST_USE_CONDA_SYSTEM=${LSST_USE_CONDA_SYSTEM:-true}
            
            
# these optional env vars may be used by functions but should be considered
            # unstable and for internal testing only.
            #
            # LSST_OS_FAMILY
            # LSST_OS_RELEASE

uses 4.1.0 and that’s what we use for all our builds at the moment. Did you start from a fresh shell (otherwise you might be picking up an environment variable override)?

plah · September 9, 2022, 2:31am

Hello Tim,

From the newinstall.sh I believe rubin-env = 4.1.0.
I started the installation in a a fresh shell and finally it completed the installation. I ran the demo and it also worked. Thanks for your help in getting to this stage.