First, a bug report:
this page, Install with newinstall.sh and eups distrib — LSST Science Pipelines
I believe incorrectly implies that “-P” takes the path to the python executable to use. This does not seem to be true – newinstall.sh seems to add “bin/conda” to the end of the “-P” argument and expects that to be the conda program. Eg,
RUN ./newinstall.sh -b -t -P /opt/conda/bin/python
---> Running in 29508bef3246
LSST Software Stack Builder
=======================================================================
Using conda at /opt/conda/bin/python
./newinstall.sh: line 672: /opt/conda/bin/python/bin/activate: Not a directory
whereas this works:
RUN ./newinstall.sh -b -t -P /opt/conda
---> Running in c560bb242f9d
LSST Software Stack Builder
=======================================================================
Using conda at /opt/conda
Collecting package metadata (current_repodata.json): ...working...
Second, my goal is to have the LSST software installed in the same conda environment as some other software, which is using the Intel Python stack (with MKL, Intel custom versions of numpy, etc). (I say “environment”, but I’m just using the “base” conda environment.) When I tell newinstall.sh to use my /opt/conda, however, it is installing its own Python, which is even a different version than the one I gave it (3.8 vs 3.7). Is there a way I can tell it, “No, really, use THIS python.”?
Okay, my other package can work with Python 3.8 instead, and it looks like the full Intel stack is available…
But when I try to use that with newinstall.sh, it seems to still want to install its own python, numpy, etc. Presumably that’s either because it’s demanding the conda-forge channel, or because it’s demanding exact versions so that my existing ones don’t work? Is there any way to work around this?
I really only need afwImage, so I’m not particularly worried about a meas package incompatibility.
@ktl will have the details but the newinstall script now requires that it has access to the rubin-env conda-forge package. We keep (some of) the conda-forge versions in that meta package pinned to ensure compatibility with a specific weekly and you will see that newinstall.sh has a specific rubin-env version that it will try to install.
Essentially we are assuming that newinstall will install this conda env and then users will install their packages on top. I’m not sure how easy it is to say to newinstall that the user should be trusted and to use whatever conda env it has been given. This might work when installing a couple of packages but, for example, we are sort of relying on the conda-forge compilers at this point so we are not really testing sconsUtils with other compilers. Also note that we pull eups in from conda-forge now.
You could try manually installing rubin-env=0.6.0 into your favorite conda environment to see what it wants to upgrade. We do allow you to “bring your own environment”, but you then become responsible for stack compatibility. I would recommend that you build from source (do not include -t or remove the binary directory from EUPSPKG_PATH) if you’re going this route.
Step 11/68 : ENV LSST_CONDA_ENV_NAME=base
---> Running in 895562ab1a87
Removing intermediate container 895562ab1a87
---> 1fab3ebf6ea3
Step 12/68 : RUN curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh && chmod 755 newinstall.sh && ./newinstall.sh -b -T -P /opt/conda
---> Running in 40fbdc82a026
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 26053 100 26053 0 0 150k 0 --:--:-- --:--:-- --:--:-- 150k
LSST Software Stack Builder
=======================================================================
Using conda at /opt/conda
CondaValueError: The target prefix is the base prefix. Aborting.
Oops, looks like the logic here needs some fixes. I’ve been working on a different script that will replace newinstall.sh, but I’ll look into patching this in the current script.
As a short-term workaround, you should be able to newinstall into the default location but then ignore that environment and use LSST_CONDA_ENV_NAME to override it when doing source loadLSST.bash. I believe this even works with LSST_CONDA_ENV_NAME=base.
The following seems to be working for me, using your newinstall.sh branch (the newinstall.sh step finishes almost instantaneously; it doesn’t try to install anything new into the conda environment!):
(this is not the whole Dockerfile; before this, I install apt packages and set up my Intel-python conda environment)
ENV BASH_ENV=/etc/bash.bashrc
RUN echo “source /opt/conda/etc/profile.d/conda.sh && conda activate myenv” > /etc/profile.d/myenv.sh
RUN chmod 755 /etc/profile.d/myenv.sh
RUN conda config --add channels conda-forge &&
conda install rubin-env=0.6.0
RUN conda install eups
ENV LSST_CONDA_ENV_NAME=myenv
ENV LSST_CONDA_CHANNELS=“intel conda-forge”
RUN curl -OL https://raw.githubusercontent.com/lsst/lsst/tickets/DM-31235/scripts/newinstall.sh &&
chmod 755 newinstall.sh &&
bash newinstall.sh -b -T -P /opt/conda
ENV EUPS_SHELL=bash
RUN rm /etc/profile.d/myenv.sh /etc/profile.d/conda.sh
RUN source /root/lsst_stack/loadLSST.bash && eups distrib install -t v22_0_1 sconsUtils
RUN source /root/lsst_stack/loadLSST.bash && eups distrib install -t v22_0_1 astshim
RUN source /root/lsst_stack/loadLSST.bash && eups distrib install -t v22_0_1 afw
Thank you for all your help here! I’ll publish this Dockerfile when it’s ready.
cheers,
–dustin
(Looks like I spoke too soon:
Step 22/76 : RUN source /root/lsst_stack/loadLSST.bash && eups distrib install -t v22_0_1 afw
---> Running in 9d17244de1c7
[ 1/12 ] sconsUtils 22.0.1 (already installed) done.
[ 2/12 ] astshim 22.0.1+01bcf6a671 (already installed) done.
[ 3/12 ] base 22.0.1+01bcf6a671 ... done.
[ 4/12 ] sphgeom 22.0.1+01bcf6a671 ... done.
[ 5/12 ] pex_exceptions 22.0.1+cc34b8281e ... done.
[ 6/12 ] utils 22.0.1+1a1dd69893 ... tests/test_pybind11.py 124 23 22 1 82%
tests/test_utils.py 167 6 4 2 95%
tests/test_wrappers.py 338 6 10 3 97%
---------------------------------------------------------------------------
TOTAL 1581 193 318 50 86%
Coverage HTML written to dir tests/.tests/pytest-utils.xml-htmlcov
Coverage XML written to file tests/.tests/pytest-utils.xml-cov-utils.xml
=========================== short test summary info ============================
FAILED tests/test_deprecated.py::DeprecatedTestCase::test_deprecate_pybind11
=================== 1 failed, 104 passed, 6 skipped in 4.50s ===================
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/utils-22.0.1+1a1dd69893/tests/.tests/pytest-utils.xml.failed
The following tests failed:
/root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/utils-22.0.1+1a1dd69893/tests/.tests/pytest-utils.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
***** error: from /root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/build.log:
eups distrib: Failed to build utils-22.0.1+1a1dd69893.eupspkg: Command:
source "/opt/conda/envs/myenv/eups/bin/setups.sh"; export EUPS_PATH="/root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0"; (/root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/build.sh) >> /root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/build.log 2>&1 4>/root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/build.msg
exited with code 252
=================================== FAILURES ===================================
__________________ DeprecatedTestCase.test_deprecate_pybind11 __________________
[gw15] linux -- Python 3.8.11 /opt/conda/envs/myenv/bin/python3.8
self = <test_deprecated.DeprecatedTestCase testMethod=test_deprecate_pybind11>
def test_deprecate_pybind11(self):
def old(x):
"""Docstring"""
return x + 1
# Use an unusual category
> old = lsst.utils.deprecate_pybind11(
old, reason="For testing.", category=PendingDeprecationWarning)
tests/test_deprecated.py:34:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
python/lsst/utils/deprecated.py:72: in deprecate_pybind11
return deprecated.sphinx.deprecated(
/opt/conda/envs/myenv/lib/python3.8/site-packages/deprecated/classic.py:267: in deprecated
adapter = adapter_cls(**kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <deprecated.sphinx.SphinxAdapter object at 0x7f36e1dda160>
directive = 'deprecated', reason = 'For testing.', version = None, action = None
category = <class 'PendingDeprecationWarning'>, line_length = 70
def __init__(
self,
directive,
reason="",
version="",
action=None,
category=DeprecationWarning,
line_length=70,
):
"""
Construct a wrapper adapter.
:type directive: str
:param directive:
Sphinx directive: can be one of "versionadded", "versionchanged" or "deprecated".
:type reason: str
:param reason:
Reason message which documents the deprecation in your library (can be omitted).
:type version: str
:param version:
Version of your project which deprecates this feature.
If you follow the `Semantic Versioning <https://semver.org/>`_,
the version number has the format "MAJOR.MINOR.PATCH".
:type action: str
:param action:
A warning filter used to activate or not the deprecation warning.
Can be one of "error", "ignore", "always", "default", "module", or "once".
If ``None`` or empty, the the global filtering mechanism is used.
See: `The Warnings Filter`_ in the Python documentation.
:type category: type
:param category:
The warning category to use for the deprecation warning.
By default, the category class is :class:`~DeprecationWarning`,
you can inherit this class to define your own deprecation warning category.
:type line_length: int
:param line_length:
Max line length of the directive text. If non nul, a long text is wrapped in several lines.
"""
if not version:
# https://github.com/tantale/deprecated/issues/40
> raise ValueError("'version' argument is required in Sphinx directives")
E ValueError: 'version' argument is required in Sphinx directives
/opt/conda/envs/myenv/lib/python3.8/site-packages/deprecated/sphinx.py:89: ValueError
- generated xml file: /root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/utils-22.0.1+1a1dd69893/tests/.tests/pytest-utils.xml -
The test failure is because you are using rubin-env 0.6.0 with Science Pipelines v22_0_1 instead of rubin-env 0.4.3.
In addition, here are some more suggestions:
You shouldn’t need to conda install eups after installing rubin-env unless you’re trying to ensure that the latest version is present, as rubin-env has always included eups.
If you are executing newinstall.sh with bash, you don’t need to chmod 755 it.
I don’t think you need to set EUPS_SHELL. It should already be set to sh, and there’s no difference between that and bash to the code.
If you eups distrib install afw, it will install all of its dependencies as well, so you shouldn’t need the other two commands.
In general, reducing the number of layers in the container is desirable, so combining RUN commands like echo and chmod may be beneficial.
Yeah, this is showing a lot of my process getting to this point, and making debugging easier. The EUPS_SHELL stuff was from me struggling for a long time to get the shell to automatically source a correct set of startup files (conda or LSST) for the various kinds of shell (login, interactive, non-interactive, sh vs bash); it’s very tricky to get that right and avoid having to source LSST.sh at the beginning of each RUN line.
(Also, since your build process is so time-consuming, I didn’t want to edit my Dockerfile too much!! )
Officially it’s the version in the appropriately tagged version of newinstall.sh: lsst/newinstall.sh at 22.0.1 · lsst/lsst · GitHub, but this should match what is in the CONDA_ENV comment in the tag file.
Unfortunately, for these early versions we did not precisely follow semantic versioning rules, so 0.6.0 is not fully substitutable for 0.4.3. We hope to do better in the future.
)