Possible documentation errors

Support
#1

I have been installing MAF using LSST’s parameters as cleanly as possible, allowing Miniconda to install as requested and all. Following all documented guidelines of how to install the MAF and how to subsequently use it, I have run into a few issues with what the documentation suggests I do and what I’ve had to actually do to get past a step.

For instance, the step to ensure that the pipeline elements are all downloaded requests following this line:

eups distrib install lsst_apps

But this line ultimately always results in “lsst_apps not found.” After searching through to see how the program was looking for versions, it was found that the source page does not list “lsst_apps” as an option but rather only “lsst.” This was a small fix but took a bit to figure out and altogether shows an error in the suggested format of installing and using the program.

Similar issues have arisen in testing the installation via LSST’s demo. On only step 3 from this page there is already an issue with “setup obs_sdss” - the program produces an error after not being able to find “obs_sdss.”

I tackled this by running:

[mnewsome@ryloth.lco.gtn lsst4]$ eups distrib install obs_sdss -t v13_0

I tried doing this instead by saying “setup obs_sdss -t v13_0” to no avail, yet the above line worked, which begs the question - why are “setup” and “eups” essentially interchangeable, but with “eups” being altogether more robust?

And finally, as was discussed in a previous thread of mine, while the above line at least ran something, it failed at [44/59] due to another boolean subtract error:

***** error: from /home/mnewsome/envs/lsst_maf/lsst4/EupsBuildDir/Linux64/afw-13.0/build.log:
File “tests/testAstropyTableViews.py”, line 124, in testFlagColumn
self.assertClose(v1[“a4”], self.catalog[“a4”])
File “/home/mnewsome/envs/lsst_maf/lsst4/Linux64/utils/13.0/python/lsst/utils/tests.py”, line 659, in assertClose
return assertFloatsAlmostEqual(*args, **kwargs)
File “/home/mnewsome/envs/lsst_maf/lsst4/Linux64/utils/13.0/python/lsst/utils/tests.py”, line 568, in assertFloatsAlmostEqual
diff = lhs - rhs
TypeError: numpy boolean subtract, the - operator, is deprecated, use the bitwise_xor, the ^ operator, or the logical_xor function instead.

I am hoping that this is fixed by instead running this code on the w_2017_23 (rather than v13) install as suggested by another user, but then again there is an issue with none of this being addressed in any documentation intended to help users install MAF.

I am posting here before submitting any tickets because I want to make sure that there are no easy explanations that I may be missing. I am new to this project so some of my concerns may be naivety, but others may be true issues with documentation errors that will only be a detriment to incoming scientists. Thanks for any help or explanations!

1 Like
#2

Which documentation are you following to install MAF and the Stack? I’ll try to retrace your steps to see why lsst_apps wasn’t being installed.

You can see what’s installed by EUPS by running

eups list

The eups command is the main program you’ll use to manage EUPS packages. The exceptions through are the setup and unsetup commands, whose roles are to “set up” and “un-set up” installed packages. By “set up” we mean activate. You can have lots of packages, and lots of versions of the same packages installed simulteneously, but only the packages that are set up are used.

In other words, eups distrib install [...] installs the package, and setup [...] lets you actually use the package.

You can see what packages are set up at any moment by running

eups list -s

(The -s means set up).

#3

I’m using the following links overall:

https://confluence.lsstcorp.org/display/SIM/Catalogs+and+MAF
https://confluence.lsstcorp.org/display/SIM/MAF+Installation+and+Initial+Examples#MAFInstallationandInitialExamples-runningexamples
And of course, https://pipelines.lsst.io/index.html

Concerning lsst_apps specifically, that was actually prompted to me after “bash newinstall.sh” was finished, so I took it to be the most direct function being suggested to me as it was straight from the source. Even after sourcing the loadLSST.bash and setting up the system, this wouldn’t work, as mentioned.

However, your answer regarding eups and setup is huuuugely helpful. Still, all the documentation told me to do was to use “setup,” so I was technically never prompted to install obs_sdss before I was told to set it up. Perhaps that is trivial, or perhaps it would not have been a problem had I been able to install lsst_apps initially the “correct” way rather than guessing another route. Not sure.

Thank you for the answer, though, it does clarify some confusion I had!

#4

Update: This is what happened upon running the same line, but with a user-suggested newer version:

[mnewsome@ryloth.lco.gtn lsst4]$ eups distrib install obs_sdss -t w_2017_23

And the same error occurred, listed below for thoroughness. I want to note that I have tried running this on my environment for Python 2.7, Py3.6, and finally by allowing the installation to run its own Miniconda Python which is running numpy v 1.13.0. If it’s a matter of needing an earlier version of numpy. it seems odd that the Miniconda run from the newinstall.sh wouldn’t already have that fixed.

***** error: from /home/mnewsome/envs/lsst_maf/lsst4/EupsBuildDir/Linux64/afw-13.0-60-gfc1c281/build.log:
self.catalog.columns[“a_flag”])
File “/home/mnewsome/envs/lsst_maf/lsst4/Linux64/utils/13.0-4-g2675101+1/python/lsst/utils/tests.py”, line 638, in assertFloatsEqual
return assertFloatsAlmostEqual(testCase, lhs, rhs, rtol=0, atol=0, **kwargs)
File “/home/mnewsome/envs/lsst_maf/lsst4/Linux64/utils/13.0-4-g2675101+1/python/lsst/utils/tests.py”, line 554, in assertFloatsAlmostEqual
diff = lhs - rhs
TypeError: numpy boolean subtract, the - operator, is deprecated, use the bitwise_xor, the ^ operator, or the logical_xor function instead.

Ran 20 tests in 0.259s

#5

I think what is going on is there is some confusion that we should address on the confluence Catalogs+MAF page. (https://confluence.lsstcorp.org/display/SIM/Catalogs+and+MAF)

Basically, the LSST simulations packages (including sims_maf … aka MAF) build upon LSST’s DM software. The “newinstall.sh” script is produced by the DM team, which is why it instructs you to “eups distrib install lsst_apps” at the end. For MAF, you need many of those packages, and DM keeps track of the general requirements for installation, so we just refer users to the basic installation instructions at pipelines.lsst.io … we should probably make some more in-depth comments about why we shuffle users there and what they can safely ignore. For MAF and for simulations, we have a package called “lsst_sims” and you can also just install “sims_maf” via those same "eups distrib install -t " instructions. For simulations, the would usually be “lsst_sims” or “sims_maf” and the tag is generally “sims” (rather than “v13”).

Some of your other questions about the demo and obs_sdss are things that are not related to sims or MAF at all, so we don’t actually address them. The demo and obs_sdss, etc., are relevant for users who want to do image processing or deal with source catalogs from images … but they are not necessary for MAF or the sims packages. I’m not sure how we can head people off from attempting to deal with these, but we can add a comment along those lines on the confluence Catalogs+MAF page.

But to address some of the specific problems you’ve had …

As @jsick said, eups is a piece of software to do package management. It works in two modes: installing a package and its dependencies, and (after packages are installed) keeping track of those installed packages and their dependencies. The command "eups distrib install -t " installs packages (and their dependencies), while the command "eups declare -r -t " lets eups know about the locations of those installed packages – usually, users do not need to worry about “eups declare” IF they are just using packages which were installed with “eups distrib install …” because it happens automatically. After packages are known to eups (either by completing the “eups distrib install” command or by using “eups declare”), then you activate them in your environment by doing "setup -t ". What eups does when you “setup” a package is make sure that all its dependencies are activated as well.

I believe python 3.6 and numpy > 1.12 have some problems with some of the simulations (and apparently the obs_sdss) packages. The stack hasn’t been thoroughly tested with these newer packages, and allowing newinstall.sh to install miniconda should currently give you a python 3.5 stack with an older numpy. I’m not sure why your LSST miniconda install ended up with numpy 1.13, this seems like it might be a bug (I could be wrong). Definitely the v13 release of lsst_apps will not work with the newest version of numpy + python 3.6, but I think that w.2017.23 should (I did not try obs_sdss and the demo, so not sure about those). Note that to get successful results with the demo, you need to be sure to use the same version as lsst_apps (i.e. the demo would have to be -t w.2017.23 as well as lsst_apps -t w.2017.23).

I suspect that we do not currently have a version of sims_maf that is tagged and built against w.2017.23 of the LSST stack, so you could not currently use “eups distrib install” to get sims_maf that will build on py3.6 … we’ll get a new release and tag put together as soon as we can verify that we have a stack that would build with these new python and numpy versions, but in the meantime it’s not really easy to use newinstall + “eups distrib install” to get this working.

Finally - on the documentation front - I think we do need somewhere some basic information about eups, how it works, what setup does, what tags are, what tags and package versions are available to install, what tags and packages are available to set up, and what packages are actually set up. I don’t think we have that available anywhere for users. I think this might be something that SQUaRE can take on perhaps, @jsick? (I have certainly seen a lot of the questions and problems that people have run into, so if you want to collaborate on it, that’s good too).

1 Like
Ideas for EUPS how-to documentation
#6

Just wanted to note that on this page, it instructs to run ipython to view and manage the tutorials, but it is apparent that instead jupyter is needed. Small fix, but should be updated. Not sure who runs that page or how to edited it but I wanted to bring it to somebody’s attention.

#7

Just wanted to say thank you for all the clarification, and I appreciate the effort gone into updating the documentation for future users!

#8

We’re kind of between places in documentation on running and using MAF, and were hoping not to invest any time in updating the confluence documentation stuff on those pages because we want to leave that behind. I think we need to figure out what to do sooner rather than later! I did update ‘ipython’ to ‘jupyter’ and corrected some of the directory names.

1 Like