I’ve just merged DM-3372, which implements RFC-169 (see also this previous discussion on CLO). This introduces some run-time mechanisms for collecting a list of packages that are being used and their versions (in the ‘base’ package) and integrates this with
CmdLineTask (in the ‘pipe_base’ package) to track and check what versions are being used in order to prevent the user from unknowingly producing data with multiple versions of the code. This is important because data produced with different versions are difficult to debug due to the uncertain provenance.
This check can be disabled (e.g., for developers who are actively changing package versions and understand the implications) using the
--no-versions command-line flag. The persisted set of package versions can be overwritten (e.g., when you’ve changed versions deliberately and understand the implications) using the
--clobber-versions command-line flag; note that
--clobber-versions is subject to race conditions (just like
--clobber-config) if you’re running it in different processes.
Please let me know if you have any trouble with this new feature.
There are a few different types of packages, and their versions are collected in different ways:
- Run-time libraries (e.g., cfitsio, fftw): we get their version from interrogating the dynamic library. (There’s a new facility in the ‘base’ package that helps with pulling symbols out of dynamic libraries.)
- Python modules (e.g., afw, numpy; galsim is also in this group even though we only use it through the library, because no version information is currently provided through the library, but that’ll change soon): we get their version from the
__version__module variable, set at build time. Note that this means that we’re only aware of modules that have already been
- Some other packages provide no run-time accessible version information (e.g., astrometry_net): we get their version from interrogating the environment. Currently, that means EUPS; if EUPS is replaced or dropped then we’ll need to consider an alternative means of getting this version information.
- Local versions of packages (a non-installed EUPS package, selected with
setup -r /path/to/package): we identify these through the environment (EUPS again) and use as a version the path supplemented with the git SHA and, if the git repo isn’t clean, an MD5 of the diff.