Lsst_build (lsstsw) changes - New asyncio-based version of lsst_build

Data Management DM Notifications
#1

With DM-26035, we will merge updates to lsst_build which should be transparent but are being documented here for users of lsst_build/lsstsw. These changes have been made to enable additional optimizations of lsst_build in the future.

tl;dr

The biggest user-facing change is asyncio-based checkouts, which means git and git-lfs checkouts are now done in parallel. As a result, they are much faster, but log messages are structured in a shuffled order. Version suffixes for eups packages are no longer sequence based but are now hash based.

Additional Info

Details of the changes include:

  • checkouts occur in parallel rather than serially.
  • The selected ref (branch) is in the logging output
  • the git-lfs fetch (the slow download part) occurs after all repos are checked out, not during the initial clone, and all LFS repos are effectively fetched in parallel, which improves bandwidth utilization. This is so we can get git information about a repo before downloading all of the LFS data, which is useful for building a manifest.
  • Manifests, and by extension builds, are in a topological sorted order. This is a consequence of using a different topological sort implementation which generates groups of dependencies that can be processed in any order. It is those groups which are sorted alphabetically.
  • Sequences for suffixes of package versions are no longer used; instead a hash of the dependencies’ commits is used for the suffix as implementation of RFC-714

The last two points can be seen in the manifest examples below, which are excepts from a manifest building afw.

In the first snippet, you find sequence-based version numbers for suffixes (if any — an omitted suffix is implicitly the first build). In the second snippet, you will see the hash-based suffix (after the + sign)

In the second snippet, you will find dependencies without dependencies are built first (sconsUtils, afwdata) and that dependencies within a group (e.g. astshim, base, and sphgeom) are in alphabetical order.

BUILD=b5030
sconsUtils                c4fbba5115c247242fa884bc828940457baae49a  20.0.0-1-gc4fbba5                        
base                      4a2362f47cddbeaa6169bf4d124e2e16ced6c532  20.0.0-4-g4a2362f                        sconsUtils
sphgeom                   d2e950ecf569e4b7d7d19eed9ff2a58315588f03  20.0.0-3-gd2e950e                        sconsUtils
afwdata                   d1c39c2f883e254a0e8fcd080721619ca496e1d3  20.0.0                                   
pex_exceptions            5a8b32307a533f330184baf2541fdc992f91826a  20.0.0+3                                 base
utils                     920eed2ca375a5ab36af36a63d6bc2c3a1d03ef1  20.0.0+3                                 base,pex_exceptions
...

BUILD=b5031
afwdata                   d1c39c2f883e254a0e8fcd080721619ca496e1d3  20.0.0                                   
sconsUtils                c4fbba5115c247242fa884bc828940457baae49a  20.0.0-1-gc4fbba5                        
astshim                   e74d1243ec454ae9114f78745dfbf1146f28f62e  20.0.0+f45b7d88f4                        sconsUtils
base                      4a2362f47cddbeaa6169bf4d124e2e16ced6c532  20.0.0-4-g4a2362f+f45b7d88f4             sconsUtils
sphgeom                   d2e950ecf569e4b7d7d19eed9ff2a58315588f03  20.0.0-3-gd2e950e+f45b7d88f4             sconsUtils
pex_exceptions            5a8b32307a533f330184baf2541fdc992f91826a  20.0.0+117106b506                        base
utils                     920eed2ca375a5ab36af36a63d6bc2c3a1d03ef1  20.0.0+95dc160249                        base,pex_exceptions
...