Failure to build daf_butler

I have been trying to install the LSST Science Pipelines and have reached an impasse.

I run “eups distrib install -t v23_0_2 lsst_distrib” and the installation fails at the daf_butler installation with the below error message:

[ 25/83 ] daf_butler g952c50c189+d18c45d440 …

***** error: from /data/mustard/plah/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/daf_butler-g952c50c189+d18c45d440/build.log:
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testRemoveRuns - boto…
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testCompositePutGetConcrete
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testBasicPutGet - bot…
FAILED tests/test_cliCmdQueryDimensionRecords.py::QueryDimensionRecordsTest::testWhere
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testDeferredCollectionPassing
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testPutTemplates - bo…
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testCompositePutGetVirtual
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testIngest - botocore…
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testGetDatasetTypes
FAILED tests/test_butler.py::S3DatastoreButlerTestCase::testPruneCollections
= 18 failed, 1160 passed, 165 skipped, 3 xfailed, 738 warnings in 89.56s (0:01:29) =
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /data/mustard/plah/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/daf_butler-g952c50c189+d18c45d440/daf_butler-g952c50c189+d18c45d440/tests/.tests/pytest-daf_butler.xml.failed
The following tests failed:
/data/mustard/plah/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/daf_butler-g952c50c189+d18c45d440/daf_butler-g952c50c189+d18c45d440/tests/.tests/pytest-daf_butler.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

  • exit -4
    eups distrib: Failed to build daf_butler-g952c50c189+d18c45d440.eupspkg: Command:
    source “/data/mustard/plah/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/eups/bin/setups.sh”; export EUPS_PATH="/data/mustard/plah/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0"; (/data/mustard/plah/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/daf_butler-g952c50c189+d18c45d440/build.sh) >> /data/mustard/plah/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/daf_butler-g952c50c189+d18c45d440/build.log 2>&1 4>/data/mustard/plah/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/daf_butler-g952c50c189+d18c45d440/build.msg
    exited with code 252

I have tried installing it on my laptop which runs 20.04.1-Ubuntu and also on a Uni computer which runs Centos. It fails both times at the same place. Any help that you can provide would be most welcome.

Please report the content of this file. It contains the actual test failure.

Hi, I believe I am having the exact same issue. The content of this file is attached.

pytest-daf_butler.xml.failed (121.5 KB)

Hello Tim,
I have attached the requested file.
pytest-daf_butler.xml.failed (122.0 KB)

Thanks. The error from the file is:

tests/test_cliCmdQueryDimensionRecords.py:78: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/lsst/daf/butler/tests/utils.py:142: in assertAstropyTablesEqual
    self.assertTrue(report_diff_values(table, expected, fileobj=diff), msg="\n" + diff.getvalue())
E   AssertionError: False is not true : 
E     a>  instrument   id physical_filter ... zenith_angle region   timespan  
E      ?                                                         --          ^
E     b>  instrument   id physical_filter ... zenith_angle region timespan [2]
E      ?                                                                   ^^^
E        ------------ --- --------------- ... ------------ ------ ------------
E        DummyCamComp 423             d-r ...         None   None None .. None
E        DummyCamComp 424             d-r ...         None   None None .. None

This is what you get when you are using a new astropy with an old pipelines release. It looks like you are using the 4.0.0 rubin-env and that environment only works for newer weekly releases. The old v23 release is based on a conda-forge rubin environment from last year and you must use that environment to build v23.

How did you get your newinstall.sh file? The instructions say to download the 23_0_2 version and using that you should get the correct environment. If you downloaded newinstall.sh from main then it will be too new for you.

I go my newinstall.sh by following the instructions on:

https://pipelines.lsst.io/install/newinstall.html

The exact command was:

curl -OL https://raw.githubusercontent.com/lsst/lsst/23.0.2/scripts/newinstall.sh

Is this the correct newinstall.sh for my enviroment?

Sorry, it looks like there was a problem in our release process and the wrong version of newinstall.sh got tagged for 23.0.2. Please use the version for 23.0.1 instead in the meantime.

Hello K-T Lim,
I tried the 23.0.1 release and it fails at the same point in the installation. I tried an earlier release (22.0.1) and the installation fails much earlier in the process.

Which conda environment did it install? (ie which version of rubin-env is it using?). The rubin-env for 23.0.1 installs should be using v0.8.1 which should pin astropy.

The code installs rubin-env=0.8.1.

Can you confirm your astropy version?

I am running astropy 5.1.

I am reminded that we don’t pin the astropy version in the rubin-env used by v23. We had back-ported the fix to support both astropy v4 and v5 but that is in the v23.0.3 release candidate that should be coming out at some point.

Is there a reason why you can’t use a newer release? v23 is derived from the codebase as it was back in October 2021. The v24.0 release is going to be based on the w_2022_28 tag so you should be safe installing that weekly and using it instead of v23.

How do I go about installing the weekly release?

Where you currently ask for v23_0_2 from eups distrib install you instead ask for w_2022_28. You will need to have the 4.x rubin-env (which is what you will get from the main branch for newinstall.sh)

I tried:

eups distrib install -t w_2022_28 lsst_distrib

It gets two steps further in the installation process before encountering an error.

The error mes

sage it outputs is:

[ 31/85 ] afw gba47b54d5d+94dc90c3ea …

***** error: from /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/build.log:
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/footprint1.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_imageHash.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_storable.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_key.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/pytest-afw.xml.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_simpleGenericMap.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/background.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_filterLabel.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/stacker.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/statistics.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/ticket1145.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_schema.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/test_functorKeys.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/image.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/rowColumnStats.failed
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/tests/.tests/ellipse.failed
36 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

  • exit -4
    eups distrib: Failed to build afw-gba47b54d5d+94dc90c3ea.eupspkg: Command:
    source “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/eups/bin/setups.sh”; export EUPS_PATH="/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0"; (/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/build.sh) >> /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/build.log 2>&1 4>/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/build.msg
    exited with code 252

I have attached the log build.log file that goes with the error.
build.log (1.5 MB)

This is the error:

tests/maskedImage

tests/maskedImage: symbol lookup error: /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/afw-gba47b54d5d+94dc90c3ea/afw-gba47b54d5d+94dc90c3ea/lib/libafw.so: undefined symbol: _Py_Dealloc

It seems that a recent change to how linking worked in the conda infrastructure exposed a bug that we fixed on DM-35600. You will have to use at least w_2022_30 to get the build to go past that.

@yusra I think we will have to back-port that ticket to the v24 release.

Hello Tim,

we seem to be slowly inching forward. I ran:

eups distrib install -t w_2022_30 lsst_distrib

and got to [ 46/84 ] before getting an error which halted installation. The exact error message is:

[ 46/84 ] meas_base gcf00a76f74+12dc3af6bc …

***** error: from /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/build.log:
Coverage XML written to file tests/.tests/pytest-meas_base.xml-cov-meas_base.xml

============================= slowest 5 durations ==============================
20.95s call tests/test_GaussianFlux.py::GaussianFluxTestCase::testMonteCarlo
2.41s call tests/test_SdssShape.py::SdssShapeTestCase::testMonteCarlo
2.15s call tests/test_ScaledApertureFlux.py::ScaledApertureFluxTestCase::testApertureTruncated
2.01s call tests/test_PsfFlux.py::PsfFluxTestCase::testMonteCarlo
1.89s call tests/test_SdssCentroid.py::SdssCentroidTestCase::testMonteCarlo
=========================== short test summary info ============================
FAILED tests/test_diaCalculationPlugins.py::TestSkewDiaPsFlux::testCalculate
================= 1 failed, 256 passed, 73 warnings in 30.70s ==================
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/meas_base-gcf00a76f74+12dc3af6bc/tests/.tests/pytest-meas_base.xml.failed
The following tests failed:
/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/meas_base-gcf00a76f74+12dc3af6bc/tests/.tests/pytest-meas_base.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

  • exit -4
    eups distrib: Failed to build meas_base-gcf00a76f74+12dc3af6bc.eupspkg: Command:
    source “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/eups/bin/setups.sh”; export EUPS_PATH="/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0"; (/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/build.sh) >> /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/build.log 2>&1 4>/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_base-gcf00a76f74+12dc3af6bc/build.msg
    exited with code 252

I have attached the log build.log file that goes with the error.
build.log (837.7 KB)

This is DM-35777. Rather than trying to pick the oldest weekly that might possibly work (to keep you as close to v23 as we can) you should pick something very new like w_2022_35 (or _36 if that’s out).

Thanks for reporting the v23 build problem. We will have to backport the butler test fix to 23.0.3 or pin astropy in that conda env.

Hello Tim,

we are slowly getting there. I ran

eups distrib install -t w_2022_36 lsst_distrib

which should be the latest weekly. The installation got further stopping at:

[ 61/86 ] meas_extensions_shapeHSM g6072bfb783+8f013e7049

The error message this time was:

[ 61/86 ] meas_extensions_shapeHSM g6072bfb783+8f013e7049 …

***** error: from /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_extensions_shapeHSM-g6072bfb783+8f013e7049/build.log:
include/lsst/meas/extensions/shapeHSM/HsmAdapter.h:39:24: error: no matching function for call to ‘galsim::ImageView::ImageView(double*&, const std::shared_ptr&, int, const int&, const galsim::Bounds&)’
39 | return galsim::ImageView(ptr, _owner, 1, stride, bounds);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from src/HsmMoments.cc:32:
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:487:9: note: candidate: ‘galsim::ImageView::ImageView(galsim::ImageAlloc&) [with T = double]’
487 | ImageView(ImageAlloc& rhs) : BaseImage(rhs) {}
| ^~~~~~~~~
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:487:9: note: candidate expects 1 argument, 5 provided
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:479:9: note: candidate: ‘galsim::ImageView::ImageView(const galsim::ImageView&) [with T = double]’
479 | ImageView(const ImageView& rhs) : BaseImage(rhs) {}
| ^~~~~~~~~
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:479:9: note: candidate expects 1 argument, 5 provided
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:469:9: note: candidate: ‘galsim::ImageView::ImageView(T*, const T*, int, const std::shared_ptr<_Tp>&, int, int, const galsim::Bounds&) [with T = double]’
469 | ImageView(T* data, const T* maxptr, int nElements, const shared_ptr& owner,
| ^~~~~~~~~
/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/include/galsim/Image.h:469:9: note: candidate expects 7 arguments, 5 provided
scons: *** [src/HsmShape.os] Error 1
scons: *** [src/HsmMoments.os] Error 1
scons: building terminated because of errors.

  • exit -4
    eups distrib: Failed to build meas_extensions_shapeHSM-g6072bfb783+8f013e7049.eupspkg: Command:
    source “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/eups/bin/setups.sh”; export EUPS_PATH="/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0"; (/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_extensions_shapeHSM-g6072bfb783+8f013e7049/build.sh) >> /home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_extensions_shapeHSM-g6072bfb783+8f013e7049/build.log 2>&1 4>/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.0/EupsBuildDir/Linux64/meas_extensions_shapeHSM-g6072bfb783+8f013e7049/build.msg
    exited with code 252

I have attached the log build.log file that goes with the error.
build.log (1.2 MB)