Newinstall.sh: I *really* want to use my own conda/python version

Support
#13

The test failure is because you are using rubin-env 0.6.0 with Science Pipelines v22_0_1 instead of rubin-env 0.4.3.

In addition, here are some more suggestions:

  • You shouldn’t need to conda install eups after installing rubin-env unless you’re trying to ensure that the latest version is present, as rubin-env has always included eups.
  • If you are executing newinstall.sh with bash, you don’t need to chmod 755 it.
  • I don’t think you need to set EUPS_SHELL. It should already be set to sh, and there’s no difference between that and bash to the code.
  • If you eups distrib install afw, it will install all of its dependencies as well, so you shouldn’t need the other two commands.
  • In general, reducing the number of layers in the container is desirable, so combining RUN commands like echo and chmod may be beneficial.
#14

Okay, so is the “CONDA_ENV” keyword the “secret decoder ring” for which version of rubin-env should work with a version of the pipeline?
https://eups.lsst.codes/stack/src/tags/v22_0_1.list

Yeah, this is showing a lot of my process getting to this point, and making debugging easier. The EUPS_SHELL stuff was from me struggling for a long time to get the shell to automatically source a correct set of startup files (conda or LSST) for the various kinds of shell (login, interactive, non-interactive, sh vs bash); it’s very tricky to get that right and avoid having to source LSST.sh at the beginning of each RUN line.

(Also, since your build process is so time-consuming, I didn’t want to edit my Dockerfile too much!! :stuck_out_tongue:)

#15

Officially it’s the version in the appropriately tagged version of newinstall.sh: lsst/newinstall.sh at 22.0.1 · lsst/lsst · GitHub, but this should match what is in the CONDA_ENV comment in the tag file.

Unfortunately, for these early versions we did not precisely follow semantic versioning rules, so 0.6.0 is not fully substitutable for 0.4.3. We hope to do better in the future.

#16

I’m finding that rubin_env 0.4.3 fails the same test case:

============================= test session starts ==============================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/utils-22.0.1+1a1dd69893, configfile: setup.cfg
plugins: xdist-2.3.0, openfiles-0.5.0, forked-1.3.0, doctestplus-0.10.1, subtests-0.5.0, flake8-1.0.7, cov-2.12.1, session2file-0.1.11
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I
gw0 [111] / gw1 [111] / gw2 [111] / gw3 [111] / gw4 [111] / gw5 [111] / gw6 [111] / gw7 [111] / gw8 [111] / gw9 [111] / gw10 [111] / gw11 [111] / gw12 [111] / gw13 [111] / gw14 [111] / gw15 [111]

...........s.................................ss..F.s..s.....s............ [ 64%]
......................................                                   [100%]
=================================== FAILURES ===================================
__________________ DeprecatedTestCase.test_deprecate_pybind11 __________________
[gw15] linux -- Python 3.8.11 /opt/conda/envs/myenv/bin/python3.8

self = <test_deprecated.DeprecatedTestCase testMethod=test_deprecate_pybind11>

    def test_deprecate_pybind11(self):
        def old(x):
            """Docstring"""
            return x + 1
        # Use an unusual category
>       old = lsst.utils.deprecate_pybind11(
            old, reason="For testing.", category=PendingDeprecationWarning)

tests/test_deprecated.py:34:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
python/lsst/utils/deprecated.py:72: in deprecate_pybind11
    return deprecated.sphinx.deprecated(
/opt/conda/envs/myenv/lib/python3.8/site-packages/deprecated/classic.py:267: in deprecated
    adapter = adapter_cls(**kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <deprecated.sphinx.SphinxAdapter object at 0x7f4d8dbdac10>
directive = 'deprecated', reason = 'For testing.', version = None, action = None
category = <class 'PendingDeprecationWarning'>, line_length = 70

    def __init__(
        self,
        directive,
        reason="",
        version="",
        action=None,
        category=DeprecationWarning,
        line_length=70,
    ):
        """
        Construct a wrapper adapter.

        :type  directive: str
        :param directive:
            Sphinx directive: can be one of "versionadded", "versionchanged" or "deprecated".

        :type  reason: str
        :param reason:
            Reason message which documents the deprecation in your library (can be omitted).

        :type  version: str
        :param version:
            Version of your project which deprecates this feature.
            If you follow the `Semantic Versioning <https://semver.org/>`_,
            the version number has the format "MAJOR.MINOR.PATCH".

        :type  action: str
        :param action:
            A warning filter used to activate or not the deprecation warning.
            Can be one of "error", "ignore", "always", "default", "module", or "once".
            If ``None`` or empty, the the global filtering mechanism is used.
            See: `The Warnings Filter`_ in the Python documentation.

        :type  category: type
        :param category:
            The warning category to use for the deprecation warning.
            By default, the category class is :class:`~DeprecationWarning`,
            you can inherit this class to define your own deprecation warning category.

        :type  line_length: int
        :param line_length:
            Max line length of the directive text. If non nul, a long text is wrapped in several lines.
        """
        if not version:
            # https://github.com/tantale/deprecated/issues/40
>           raise ValueError("'version' argument is required in Sphinx directives")
E           ValueError: 'version' argument is required in Sphinx directives

/opt/conda/envs/myenv/lib/python3.8/site-packages/deprecated/sphinx.py:89: ValueError
- generated xml file: /root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/utils-22.0.1+1a1dd69893/tests/.tests/pytest-utils.xml -

---------- coverage: platform linux, python 3.8.11-final-0 -----------
Name                                      Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------
python/lsst/__init__.py                       3      0      0      0   100%
python/lsst/utils/__init__.py                 9      0      0      0   100%
python/lsst/utils/backtrace/__init__.py       1      0      0      0   100%
python/lsst/utils/deprecated.py              17      5      0      0    71%
python/lsst/utils/doImport.py                30      2      8      0    95%
python/lsst/utils/get_caller_name.py         20      0     10      2    93%
python/lsst/utils/inheritDoc.py               9      8      2      0     9%
python/lsst/utils/tests.py                  295     82    132     13    73%
python/lsst/utils/version.py                  6      0      0      0   100%
python/lsst/utils/wrappers.py               142     16     86     15    86%
tests/import_test/__init__.py                 0      0      0      0   100%
tests/import_test/two/three/fail.py           3      2      0      0    33%
tests/import_test/two/three/runtime.py        1      0      0      0   100%
tests/import_test/two/three/success.py        5      0      0      0   100%
tests/test_backtrace.py                      24      2      4      2    86%
tests/test_cache.py                          68     13     18      3    79%
tests/test_decorators.py                     33      4      6      1    87%
tests/test_deprecated.py                     14      6      2      1    56%
tests/test_doImport.py                       49      1      2      1    96%
tests/test_executables.py                    18      5      2      1    70%
tests/test_getPackageDir.py                  22      2      2      1    88%
tests/test_getTempFilePath.py                90      2      2      1    97%
tests/test_get_caller_name.py                37      2      2      1    92%
tests/test_ordering.py                       17      3      2      1    79%
tests/test_pySharedPtr.py                    39      3      2      1    90%
tests/test_pybind11.py                      124     23     22      1    82%
tests/test_utils.py                         167      6      4      2    95%
tests/test_wrappers.py                      338      6     10      3    97%
---------------------------------------------------------------------------
TOTAL                                      1581    193    318     50    86%
Coverage HTML written to dir tests/.tests/pytest-utils.xml-htmlcov
Coverage XML written to file tests/.tests/pytest-utils.xml-cov-utils.xml

=========================== short test summary info ============================
FAILED tests/test_deprecated.py::DeprecatedTestCase::test_deprecate_pybind11
=================== 1 failed, 104 passed, 6 skipped in 4.35s ===================
Global pytest run: failed with 1
Failed test output:
Global pytest output is in /root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/utils-22.0.1+1a1dd69893/tests/.tests/pytest-utils.xml.failed
The following tests failed:
/root/lsst_stack/stack/miniconda3-py38_4.9.2-0.6.0/EupsBuildDir/Linux64/utils-22.0.1+1a1dd69893/utils-22.0.1+1a1dd69893/tests/.tests/pytest-utils.xml.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.

# conda list | grep rubin
rubin-env                 0.4.3                ha770c72_0    conda-forge
#17

Ouch. What version of deprecated do you currently have in your environment?

#18 
# conda list | grep deprecated
deprecated                1.2.12             pyh44b312d_0    conda-forge
#19

That shouldn’t be possible; rubin-env 0.4.3 specifies deprecated <1.2.12. rubinenv-feedstock/meta.yaml at v0.4.3 · conda-forge/rubinenv-feedstock · GitHub

#20

The rubin-env install didn’t seem to go exactly as planned:

Step 11/77 : RUN conda config --add channels conda-forge &&     conda install rubin-env=0.4.3
 ---> Running in abe476e32ddc
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working...
done
#21

Oh. You will need a newer build of rubin-env 0.4.3. Not sure how you still got ha770c72_0, but ha770c72_4 is current.

“failed with initial frozen solve” is not usually a problem.

#22

I just ran

RUN conda config --add channels conda-forge && \
    conda install rubin-env=0.4.3

like 10 minute ago, so

#23

So going back a step, what version of conda are you using?

#24 
RUN wget -nv https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh && \
    bash miniconda.sh -b -p /opt/conda && \
    conda config --add channels intel && \
    conda config --set always_yes yes && \
    conda update conda && \
    conda create -n myenv python=3.8 pip numpy scipy && \
    conda clean -a -y

# conda --version
conda 4.10.3
#25

That last script seems a bit iffy as I don’t think you’re activating the downloaded conda before executing commands. With a fresh miniconda and environment, even with the intel channel and (unnecessary) pre-installation of pip/numpy/scipy, I can’t reproduce your issue, as I get:

(myenv) lsst-devl02:~/stack $ conda list deprecated
# packages in environment at /home/ktlim/stack/conda/envs/myenv:
#
# Name                    Version                   Build  Channel
deprecated                1.2.10             pyh9f0ad1d_0    conda-forge
(myenv) lsst-devl02:~/stack $ conda list rubin-env
# packages in environment at /home/ktlim/stack/conda/envs/myenv:
#
# Name                    Version                   Build  Channel
rubin-env                 0.4.3                ha770c72_4    conda-forge
#26

Sorry, what do you mean by “that last script”? I’m activating the environment in an /etc/profile.d file.

#27

I’m not sure your profile.d got a chance to run in the middle of this sequence of commands:

#28

I previously added /opt/conda/bin to PATH …

What do you think is going wrong? That ‘conda’ is some different version of conda? (there isn’t)

PS my Dockerfile is
[legacypipe/Dockerfile at lsst · legacysurvey/legacypipe · GitHub]

PPS I’m not a conda expert at all – I don’t really understand how I can have rubin-env of the desired version from the desired channel but not the correct build.

#29

Fwiw, my conda knows about the different builds but for some reason chose the first one

# conda search rubin-env=0.4.3
Loading channels: done
# Name                       Version           Build  Channel
rubin-env                      0.4.3      ha770c72_0  conda-forge
rubin-env                      0.4.3      ha770c72_1  conda-forge
rubin-env                      0.4.3      ha770c72_2  conda-forge
rubin-env                      0.4.3      ha770c72_3  conda-forge
rubin-env                      0.4.3      ha770c72_4  conda-forge

# conda list rubin-env
# packages in environment at /opt/conda/envs/myenv:
#
# Name                    Version                   Build  Channel
rubin-env                 0.4.3                ha770c72_0    conda-forge
#30

If I tell conda to install that exact build:

# conda install rubin-env=0.4.3=ha770c72_4
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/myenv

  added / updated specs:
    - rubin-env==0.4.3=ha770c72_4


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    click-7.1.2                |     pyh9f0ad1d_0          64 KB  conda-forge
    deprecated-1.2.10          |     pyh9f0ad1d_0          11 KB  conda-forge
    galsim-2.2.5               |   py38h3878127_0         3.0 MB  conda-forge
    pandas-1.2.5               |   py38h1abd341_0        12.1 MB  conda-forge
    rubin-env-0.4.3            |       ha770c72_4           7 KB  conda-forge
    sqlite-3.34.0              |       h74cdb3f_0         1.4 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        16.6 MB

The following packages will be UPDATED:

  rubin-env                                0.4.3-ha770c72_0 --> 0.4.3-ha770c72_4

The following packages will be SUPERSEDED by a higher-priority channel:

  click              conda-forge/linux-64::click-8.0.1-py3~ --> conda-forge/noarch::click-7.1.2-pyh9f0ad1d_0
  sqlite                    intel::sqlite-3.35.5-hb9a14ef_1 --> conda-forge::sqlite-3.34.0-h74cdb3f_0

The following packages will be DOWNGRADED:

  deprecated                            1.2.12-pyh44b312d_0 --> 1.2.10-pyh9f0ad1d_0
  galsim                               2.3.2-py38h260733e_1 --> 2.2.5-py38h3878127_0
  pandas                               1.3.1-py38h1abd341_0 --> 1.2.5-py38h1abd341_0



Downloading and Extracting Packages
pandas-1.2.5         | 12.1 MB   | ############################################################################################################################################################### | 100%
galsim-2.2.5         | 3.0 MB    | ############################################################################################################################################################### | 100%
click-7.1.2          | 64 KB     | ############################################################################################################################################################### | 100%
sqlite-3.34.0        | 1.4 MB    | ############################################################################################################################################################### | 100%
deprecated-1.2.10    | 11 KB     | ############################################################################################################################################################### | 100%
rubin-env-0.4.3      | 7 KB      | ############################################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: sed: command not found
bash: grep: command not found

Those final complaints are… weird… as though the PATH has gotten messed up, but once the command completes, my shell path is fine -

# which sed
/bin/sed
# which grep
/bin/grep
#31

I’ve merged DM-31235, so the master version of newinstall.sh should now accept a specified conda environment if it already exists without installing anything.

#32

Unfortunately this caused problems with building our binary tarballs, so I had to back it out. I will reopen the ticket and investigate further how to do this properly.