Procedure for adding a third-party Python dependency as a pseudo EUPS package (?)

Data Management
#1

I want to add jsonschema as a third-party python dependency of validate_base (we’ll have an announcement post about that soon) as part of DM-7935. I’m a bit unclear about our handling of python externals, and the Developer Guide page Distributing Third-Party Packages with EUPS doesn’t entirely fit my scenario.

First, it seems that there are two ways we handle python externals in the Stack:

  1. As TaP packages, like those described in Distributing Third-Party Packages with EUPS that contains the python package as a tar file.
  2. As an EUPS ‘pseudo package’ that merely checks that the package is installed. An example is https://github.com/lsst/astropy
  3. Are there other alternatives?

I think my preference is to go with #2 so we’re not replicating what PyPI does already. Is this a correct assumption?

Going with #2, the procedure would then be:

  1. Create a repo https://github.com/lsst/jsonschema and implement it like https://github.com/lsst/astropy.
  2. RFC this jsonschema package. (I’m not 100% sure this is necessary as it will only a dependency of the lsst_qa top-level package; not lsst_apps or lsst_distrib. Maybe @timj can advise.).
  3. Publish the jsonschema package following https://developer.lsst.io/build-ci/third_party.html#distributing-the-package.
  4. Add the jsonschema package to lsstsw’s repos.yaml.
  5. Add jsonschema to the conda package lists in https://github.com/lsst/lsstsw/tree/master/etc.
  6. Document the dependency in https://pipelines.lsst.io/install/newinstall.html: pip install jsonschema==2.5.1.
  7. Declare the dependency in validate_base.

I think steps 5, 6 and 7 are what I’m most unsure of. Is anything else needed to ensure that the jsonschema python package is installed for newinstall, lsstsw and conda stack users?

Alternatively, should a jsonschema eups pseudopackage actually run pip install setup.py through its ups/eupspkg.cfg.sh.

I’ll turn responses to this topic and a corrected procedure into improved Developer Guide documentation.

Is scons used by lsstsw to install third-party lightly-warpped packages?
#2

It partly depends on how many dependencies the python package has. If it has no dependencies itself then I think I’d edge towards a full EUPS package of the tar file as that gives you much more control over versioning.

A two day RFC wouldn’t hurt.

Would also need to add it to lsstsw/bin/deploy so it gets installed in bleeding edge mode (if you are going for option #2).

#3

Looks like jsonschema has only one dependency at setup time, though it’s possibly not necessary for the TaP package. Maybe you’re right that a full TaP package is the way to go to reduce the end-user’s installation burden.

Thanks for checking on the procedure for making the alternative TaP package.