Over the winter break I have modified package installation with SCons (specifically sconsUtils) such that the science pipelines packages now conform to Python packaging metadata standards.
As of w_2025_01 package versions can now be obtained using importlib.metadata:
>>> from importlib.metadata import version
>>> version("lsst.daf.butler")
'g1ba37b057b+b0ba254fc4'
This works for lsst-daf-butler or lsst.daf.butler but it does not work with the EUPS name of daf_butler. There is no longer a need to use the (deprecated by Python tooling) __version__ package variable, although that is still set by our pipelines packages. This new functionality is achieved by creating a <package>.dist-info directory in the package’s python/ directory containing a standard METADATA file.
One additional feature of this change is that all our pipeline packages now support standard Python entry points defined in the pyproject.toml file (in the project.entry-points section). These can be used for the discovery of plugins and the middleware tools have been modified to use entry points for the butler command-line (such that it is no longer required to set the $DAF_BUTLER_PLUGINS environment variable).
To provide consistency between pip installable versions of packages and SCons-installed versions of packages, I modified sconsUtils to look for project.scripts entries in a pyproject.toml file. If any are found a python script will be automatically generated in the bin/ directory. This allows the removal of the bin.src directory for cases where the scripts conform to the recommended approach of being a thin-wrapper around a function in the main python distribution.
Furthermore, coming soon I am adding support for the RECORD metadata file (listing all the files associated with the Python package). This will allow the EUPS Python packages installed by SCons to appear as real distributions with the packages_distributions() command:
>>> from importlib.metadata import packages_distributions
>>> packages_distributions()["lsst"]
['lsst-sconsUtils', 'lsst-analysis-ap', 'lsst-display-astrowidgets', 'lsst-drp-pipe', 'lsst-atmospec', 'lsst-cp-verify', 'lsst-faro', 'lsst-fgcmcal', 'lsst-display-matplotlib', 'lsst-display-firefly', 'lsst-ap-verify', 'lsst-jointcal', 'lsst-cbp', 'lsst-obs-fiberspectrograph', 'lsst-obs-rubinGenericCamera', 'lsst-obs-cfht', 'lsst-multiprofit', 'lsst-multiprofit', 'lsst-rucio-register', 'lsst-ctrl-bps-parsl', 'lsst-ctrl-bps-panda', 'lsst-lsst-bps-plugins', 'lsst-ctrl-platform-s3df', 'lsst-ctrl-bps', 'lsst-ctrl-bps-htcondor', 'lsst-ctrl-execute', 'lsst-display-ds9', 'lsst-meas-extensions-simpleShape', 'lsst-meas-transiNet', 'lsst-source-injection', 'lsst-cell-coadds', 'lsst-drp-tasks', 'lsst-obs-lsst', 'lsst-ctrl-mpexec', 'lsst-obs-decam', 'lsst-obs-subaru', 'lsst-cp-pipe', 'lsst-analysis-tools', 'lsst-ap-association', 'lsst-ap-pipe', 'lsst-scarlet-lite', 'lsst-meas-extensions-scarlet', 'lsst-meas-extensions-psfex', 'lsst-meas-extensions-piff', 'lsst-meas-extensions-gaap', 'lsst-meas-extensions-photometryKron', 'lsst-meas-extensions-convolved', 'lsst-ip-isr', 'lsst-verify-metrics', 'lsst-sdm-schemas', 'lsst-dax-apdb', 'lsst-verify', 'lsst-meas-extensions-shapeHSM', 'lsst-meas-extensions-trailedSources', 'lsst-meas-astrom', 'lsst-ip-diffim', 'lsst-coadd-utils', 'lsst-pipe-tasks', 'lsst-shapelet', 'lsst-meas-modelfit', 'lsst-obs-base', 'lsst-pipe-base', 'lsst-daf-relation', 'lsst-resources', 'lsst-daf-butler', 'lsst-skymap', 'lsst-meas-base', 'lsst-meas-algorithms', 'lsst-log', 'lsst-sphgeom', 'lsst-geom', 'lsst-pex-config', 'lsst-pex-exceptions', 'lsst-base', 'lsst-cpputils', 'lsst-daf-base', 'lsst-afw', 'lsst-meas-deblender', 'ts_xml']
Developers will, in some cases, see a python/<package>.dist-info directory when they do git status. Please do not add this directory to git, and instead add python/*.dist-info to the .gitignore file if you encounter it. I have changed about half of the pipeline packages to ignore this directory but not all. We hope the remainder can be fixed incrementally.