Building lsst_distrib v12_0_rc1 asks for github

Support
,
#1

Hi,

Trying to build v12_0_rc1:

eups distrib install -t v12_0_rc1 lsst_distrib

At the step of building daf_butlerUtils it keeps asking github:

[ 1/90 ] apr 1.5.2 (already installed) done.
[ 2/90 ] cfitsio 3360.lsst4 (already installed) done.
[ 3/90 ] doxygen 1.8.5.lsst1 (already installed) done.
[ 4/90 ] eigen 3.2.5.lsst1 (already installed) done.
[ 5/90 ] fftw 3.3.4.lsst2 (already installed) done.
[ 6/90 ] gsl 1.16.lsst3 (already installed) done.
[ 7/90 ] mariadb 10.1.11.lsst2 (already installed) done.
[ 8/90 ] mariadbclient 10.1.11.lsst2 (already installed) done.
[ 9/90 ] minuit2 5.34.14 (already installed) done.
[ 10/90 ] mpich 3.2 (already installed) done.
[ 11/90 ] python 0.0.3 (already installed) done.
[ 12/90 ] python_d2to1 0.2.12.lsst1 (already installed) done.
[ 13/90 ] swig 3.0.2.lsst1 (already installed) done.
[ 14/90 ] xpa 2.1.15.lsst3 (already installed) done.
[ 15/90 ] activemqcpp 3.9.0.lsst2+3 (already installed) done.
[ 16/90 ] apr_util 1.5.4 (already installed) done.
[ 17/90 ] astropy 0.0.1.lsst1-2-g47e80e2 (already installed) done.
[ 18/90 ] boost 1.60 (already installed) done.
[ 19/90 ] mpi 0.0.1+1 (already installed) done.
[ 20/90 ] mysqlpython 1.2.3.lsst2 (already installed) done.
[ 21/90 ] numpy 0.0.2 (already installed) done.
[ 22/90 ] python_psutil 4.1.0 (already installed) done.
[ 23/90 ] pyyaml 3.11.lsst1 (already installed) done.
[ 24/90 ] scons 2.3.5 (already installed) done.
[ 25/90 ] sqlalchemy 1.0.8.lsst3 (already installed) done.
[ 26/90 ] stsci_distutils 0.3.7.lsst1 (already installed) done.
[ 27/90 ] wcslib 5.13.lsst1 (already installed) done.
[ 28/90 ] astrometry_net 0.50.lsst3 (already installed) done.
[ 29/90 ] esutil 0.5.3 (already installed) done.
[ 30/90 ] log4cxx 0.10.0.lsst6+1 (already installed) done.
[ 31/90 ] matplotlib 0.0.2 (already installed) done.
[ 32/90 ] mpi4py 1.3.1.lsst2 (already installed) done.
[ 33/90 ] pyfits 3.4.0+3 (already installed) done.
[ 34/90 ] scipy master-gaedd72728b (already installed) done.
[ 35/90 ] scisql 0.3.5+12 (already installed) done.
[ 36/90 ] sconsUtils 2016_01.0-8-ge052bd8 (already installed) done.
[ 37/90 ] tmv 0.73 (already installed) done.
[ 38/90 ] astrometry_net_data 10.0+72 (already installed) done.
[ 39/90 ] base 2.2016.10-2-ge90c490+1 (already installed) done.
[ 40/90 ] galsim 1.3.2.lsst1-1-g7bce591+1 (already installed) done.
[ 41/90 ] geom 10.0+58 (already installed) done.
[ 42/90 ] lmfit 0.9.3 (already installed) done.
[ 43/90 ] lsst 2016_01.0-10-g80ea1a1+2 (already installed) done.
[ 44/90 ] psfex 2016_01.0+8 (already installed) done.
[ 45/90 ] log 2016_01.0-4-g03cea0f+7 (already installed) done.
[ 46/90 ] ndarray 2016_01.0-1-g4d828c7+5 (already installed) done.
[ 47/90 ] pex_exceptions 2016_01.0-1-gfde6942+1 (already installed) done.
[ 48/90 ] db 2016_02+10 (already installed) done.
[ 49/90 ] utils 2016_01.0-5-gf0ccc9b+1 (already installed) done.
[ 50/90 ] daf_base 2016_01.0-2-g54642da+4 (already installed) done.
[ 51/90 ] ctrl_events 2016_01.0-4-gfa44e16+6 (already installed) done.
[ 52/90 ] pex_logging 2016_01.0-3-g1a9be9b+4 (already installed) done.
[ 53/90 ] pex_policy 2016_01.0-1-g9f41d6c+7 (already installed) done.
[ 54/90 ] daf_persistence 2016_01.0-9-ga3a0b07+9 (already installed) done.
[ 55/90 ] pex_config 2016_01.0-1-g6fbf654+11 (already installed) done.
[ 56/90 ] afw 2.2016.10-23-g120d329 (already installed) done.
[ 57/90 ] cat 2016_01.0-3-g7ce2ae3+34 (already installed) done.
[ 58/90 ] ctrl_provenance 2016_01.0+44 (already installed) done.
[ 59/90 ] display_ds9 2015_10.0+94 (already installed) done.
[ 60/90 ] shapelet 2016_01.0-5-g46e3358+4 (already installed) done.
[ 61/90 ] skymap 2016_01.0-2-g101aa9a+14 (already installed) done.
[ 62/90 ] skypix 10.0+400 (already installed) done.
[ 63/90 ] ctrl_orca 2016_01.0+46 (already installed) done.
[ 64/90 ] daf_butlerUtils 2016_01.0-7-g18e962e … Username for ‘https://github.com’:

Why is it so?

Christian

#2

I don’t know but will try to find out. Can you not use a binary install from Conda?

#3

I’m guessing this is because the daf_butlerUtils repository no longer exists (it has become obs_base).

If the repository had simply been renamed, I think GitHub would put in an automatic redirect and everything might keep working. But because there was an elaborate procedure of forking and moving (see DMTN-027) I guess that doesn’t work. This might mean we should revisit that procedure, or it might mean I’m confused. @parejkoj might have thoughts.

For 12.0rc1? I’m assuming the version is important, since we’re being specific about it.

#4

This is the problem that we discussed on Slack last week. lsst_distrib is looking to github for the package, when it should have a specific version already pulled down. I think @frossie made a fix for that with v12_1_2:

https://lsstc.slack.com/archives/dm/p1479251231002342

#5

Two questions arise:

  • Could a modification of the DMTN-027 procedure ensure that redirects remain in place from the old repository name to the new? I understand that GitHub will create them automatically if a repository is simply renamed or moved. Even if Frossie has worked around this particular problem for the future, it seems likely that having a transparent redirect would be a generally useful thing. (Or would it cause problems in some circumstances?)

  • Frossie’s fix means this problem won’t happen in future, but our older releases (like 12.0rc1, apparently) are still broken. Could we use some GitHub gymnastics to fix them? For example, if we quickly renamed obs_base to daf_butlerUtils then back again, would that create a redirect which might resolve @ChristianArnault’s problem?

#6

I am confused as to why v12_0_rc1 is being requested rather than v12_0. Looking on https://sw.lsstcorp.org/eupspkg/products/ it seems that

was one of the few publishes that were done “incorrectly” using git packaging. On the other hand v12_0 was published correctly as a tar file. Do not use v12_0rc1.

As @parejkoj says, v12_1_2 will also work and is more recent.

#7

Hi

Thanks for all these quite detailed explanation.

Just two words on why I tried to install this release:

I was in the process of automating the regular installation of releases (“weeklies”). [Following the idea of John’s shared_stack tool].

Of course, while getting the automatic list of tags, I got this one.
I understand now that this release is “bad”, and after all it’s not required by our users here at CCINP23.

=> It’s straighforward for me to “manually” discard one of the detected releases.

But wouldn’t be usefull to tag “bad” releases (with a known pattern) so as to discard them during automatic listing?

Regards
Christian

#8

What command are you using to list releases? eups distrib doesn’t try to interpret release tags and if you look in the server directory you’ll see that there are many different historical approaches for tagging. @frossie or @ktl may have an opinion as to whether we should clear out the history (and in particular whether release candidates should have a shelf life on the server).

#9

Hi

I’m first querying about all tags matching a given pattern (eg. v12_* ) using the following http request:

https://sw.lsstcorp.org/eupspkg/tags

Then I use the newinstall.sh script followed by a sequence of

eups distrib install -t "$VERSION" lsst_distrib

calls for each tag previously selected.

Christian

#10

@ChristianArnault the weekly releases are tagged w_YYYY_WW if that helps.

Also, are you interested in weekly docker containers instead? That should be more efficient with you. We’re just done with the production process and about to announce their availability.

#11

Hi

Very good news!
Yes this will an improvement

However, I’d like to understand the difference (if any) between the procedure

https://developer.lsst.io/build-ci/lsstsw.html

and the one described in:

https://confluence.lsstcorp.org/display/LSWUG/Building+the+LSST+Stack+from+Source

Which is recommended and why?

Christian

#12

lsstsw is the build tool that tends to be more favored by stack developers as it uses the git repositories directly and will build the most recent version of the software.

newinstall plus eups distrib is used to build tagged releases, and is the scheme most favored by external users wanting source distributions.

Note that confluence pages are deprecated. The page you probably want for newinstall is https://pipelines.lsst.io/install/newinstall.html