With DM-26035, we will merge updates to
lsst_build which should be transparent but are being documented here for users of
lsstsw. These changes have been made to enable additional optimizations of
lsst_build in the future.
The biggest user-facing change is
asyncio-based checkouts, which means
git-lfs checkouts are now done in parallel. As a result, they are much faster, but log messages are structured in a shuffled order. Version suffixes for
eups packages are no longer sequence based but are now hash based.
Details of the changes include:
- checkouts occur in parallel rather than serially.
- The selected ref (branch) is in the logging output
git-lfsfetch (the slow download part) occurs after all repos are checked out, not during the initial clone, and all LFS repos are effectively fetched in parallel, which improves bandwidth utilization. This is so we can get
gitinformation about a repo before downloading all of the LFS data, which is useful for building a manifest.
- Manifests, and by extension builds, are in a topological sorted order. This is a consequence of using a different topological sort implementation which generates groups of dependencies that can be processed in any order. It is those groups which are sorted alphabetically.
- Sequences for suffixes of package versions are no longer used; instead a hash of the dependencies’ commits is used for the suffix as implementation of RFC-714
The last two points can be seen in the manifest examples below, which are excepts from a manifest building
In the first snippet, you find sequence-based version numbers for suffixes (if any — an omitted suffix is implicitly the first build). In the second snippet, you will see the hash-based suffix (after the
In the second snippet, you will find dependencies without dependencies are built first (
afwdata) and that dependencies within a group (e.g.
sphgeom) are in alphabetical order.
BUILD=b5030 sconsUtils c4fbba5115c247242fa884bc828940457baae49a 20.0.0-1-gc4fbba5 base 4a2362f47cddbeaa6169bf4d124e2e16ced6c532 20.0.0-4-g4a2362f sconsUtils sphgeom d2e950ecf569e4b7d7d19eed9ff2a58315588f03 20.0.0-3-gd2e950e sconsUtils afwdata d1c39c2f883e254a0e8fcd080721619ca496e1d3 20.0.0 pex_exceptions 5a8b32307a533f330184baf2541fdc992f91826a 20.0.0+3 base utils 920eed2ca375a5ab36af36a63d6bc2c3a1d03ef1 20.0.0+3 base,pex_exceptions ...
BUILD=b5031 afwdata d1c39c2f883e254a0e8fcd080721619ca496e1d3 20.0.0 sconsUtils c4fbba5115c247242fa884bc828940457baae49a 20.0.0-1-gc4fbba5 astshim e74d1243ec454ae9114f78745dfbf1146f28f62e 20.0.0+f45b7d88f4 sconsUtils base 4a2362f47cddbeaa6169bf4d124e2e16ced6c532 20.0.0-4-g4a2362f+f45b7d88f4 sconsUtils sphgeom d2e950ecf569e4b7d7d19eed9ff2a58315588f03 20.0.0-3-gd2e950e+f45b7d88f4 sconsUtils pex_exceptions 5a8b32307a533f330184baf2541fdc992f91826a 20.0.0+117106b506 base utils 920eed2ca375a5ab36af36a63d6bc2c3a1d03ef1 20.0.0+95dc160249 base,pex_exceptions ...