Ndarray tests failing in hscpipe8/lsst pipeline installation

Support
#1

Hi all, I’m trying to download the HSC pipeline, hscpipe8, which I understand builds off of LSST analysis tools. In particular, I have been following the instructions here to build from source: hscPipe8 installation — hscPipe8 manual

And the relevant installation step I’m on is: eups distrib install hscPipe 8.4

The process has been rocky so far, and now there seems to be an issue building ndarray. The gist of the error is that none of the tests are working; I’ve pasted the full error output at the end of this post. I’ve been having some trouble parsing this issue, and haven’t found good leads elsewhere, so I was hoping there might be folks here who can help diagnose the issue etc. Appreciate in advance any consideration you can give to this!

  [ 43/94 ]  ndarray 8.0-hsc ... 

***** error: from /global/scratch/projects/co_dweisz/swfu/hscpipe8/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ndarray-8.0-hsc/build.log:
    Start 3: test_ndarray_eigen
3/5 Test #3: test_ndarray_eigen ...............***Failed    0.00 sec
    Start 4: test_ndarray-fft
4/5 Test #4: test_ndarray-fft .................***Failed    0.00 sec
    Start 5: pybind11_test
5/5 Test #5: pybind11_test ....................***Failed    0.61 sec

0% tests passed, 5 tests failed out of 5

Total Test time (real) =   0.68 sec

The following tests FAILED:
	  1 - test_ndarray (Failed)
	  2 - test_views (Failed)
	  3 - test_ndarray_eigen (Failed)
	  4 - test_ndarray-fft (Failed)
	  5 - pybind11_test (Failed)
Errors while running CTest
make: *** [test] Error 8
+ exit -4
eups distrib: Failed to build ndarray-8.0-hsc.eupspkg: Command:
	source "/global/scratch/projects/co_dweisz/swfu/hscpipe8/eups/2.1.5/bin/setups.sh"; export EUPS_PATH="/global/scratch/projects/co_dweisz/swfu/hscpipe8/stack/miniconda3-4.7.10-4d7b902"; (/global/scratch/projects/co_dweisz/swfu/hscpipe8/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ndarray-8.0-hsc/build.sh) >> /global/scratch/projects/co_dweisz/swfu/hscpipe8/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ndarray-8.0-hsc/build.log 2>&1 4>/global/scratch/projects/co_dweisz/swfu/hscpipe8/stack/miniconda3-4.7.10-4d7b902/EupsBuildDir/Linux64/ndarray-8.0-hsc/build.msg 
exited with code 252
#2

Hi @swfu,

I don’t have personal experience with installing the HSC pipeline, but found a relevant post. I suggest to review it and try what Dustin did, which can be found at the end of the conversation.

#3

Unless you’re trying to replicate old results obtained by running hscPipe, I would strongly recommend using the latest LSST pipeline instead, which works fine (and is now better than the old hscPipe) on HSC images.

If you absolutely must install hscPipe, please try version 8.5.3 (as suggested by @galaxyumi331). Although that’s the most recent version, you may still have difficulty installing it from source because it was released in April 2021 and some third-party software providers may not be hosting their source code any more. I recommend installing from binaries if that’s possible. If not, another option is to try the (old!) Docker image paprice/hscpipe.

#4

Hi @swfu. Have you had a chance to try the suggestions by @galaxyumi331 and @price?

#5

Hi @gfonsecaalvarez et al- thanks for following up! Apologies about the delay- I had been traveling this past week for a conference and wasn’t able to revisit the issue until now. My impression from conversations with some other LSST folks is that using the LSST pipeline might be the way to go. I’ll give installation a try and report back on how things pan out!

#6

Hi all- wanted to write with an update. I was able to install both LSSTpipe and hscpipe8.4 on my cluster, and for my purposes I figured it might be useful to have both anyway. In case it might be useful:

For LSST, I was following these instructions, and only thing of note is that I was prompted to run newinstall.sh using the latest version. As of yesterday it required running eups distrib install -t v27_0_0 lsst_distrib to install without a problem.

For HSCpipe, I can only speculate on what finally worked. Maybe it is because the OS on my cluster updated quite recently; the other is not loading any python modules before proceeding with the installation steps (probably would also apply to the above?). Otherwise I was just following the instructions here.

Thanks all for weighing in!