As you may know, we’ve been building a new approach to documenting the LSST Science Pipelines (the DMTN-030 plan) and LSST’s EUPS-based products in general. In this plan, we’re writing docstrings in the Numpydoc format, writing guides in reStructuredText, and building the pipelines.lsst.io Sphinx site as part of our Jenkins CI continuous integration pipelines.
We’re now on the verge of flipping the switch and fully adopting this new documentation system. This post will get you up to speed on the latest developments and our next steps.
New documentation on contributing documentation
Today I’ve merged a brand new set of documentation in the DM Developer Guide’s Stack section on how to contribute documentation to pipelines.lsst.io.
If you’re a DM developer writing documentation, you’ll want to look at the documentation on the layout of doc/ directories and page templates (called topic types). Our goal with these topic types is to give you starting points for documentation that make it easier to contribute documentation, make it easier to organize and maintain the documentation, and provide a more consistent experience for our readers. These templates also appear in our Stack package template project.
You’ll also want to learn about how to build documentation, both for single packages with the new package-docs tool and for the entire pipelines.lsst.io site with the stack-docs tool.
If you want a broader look at how the documentation system works, there’s an Overview of the Stack documentation system.
Recent changes to the templates and documentation build system
If you’ve participated in the doc directory upgrade and numpydoc conversion activities, you’ve already got some experience writing and building documentation in the new system. We’ve improved upon the templates and tooling you’ve used, so I want to summarize the key changes.
New build tools
The stack-docs and package-docs tools replaced the build-stack-docs tool and the direct use of sphinx-build. These tools are implemented with the terrific Click package and include both build and clean subcommands. Again, the Developer Guide has tutorials on how to use both of these tools.
Most package documentation directories are going away
A package documentation directory is something like doc/pipe_base, as opposed to a module documentation directory like doc/lsst.pipe.base. We no longer use package documentation directories for packages that provide Python modules or C++ namespaces.
We made this change based on feedback that the packaging system can be confusing to non-developers. Since most users will be working with documentation related to the Python module (see the Module homepage), it makes sense to provide links to the associated GitHub repo and Jira component from there. Over the new few days I’ll be PRing these changes to our doc/ directories.
Automodapi
We updated the recommended usage of automodapi. See the template for the module homepage. By default, we recommend using the no-main-docstr and no-inheritance-diagram options. You’re welcome to still use inheritance diagrams in cases where it’s useful.
No more doc/_static directories
The doc/_static directory is no longer required. In fact, you probably don’t need it for your packages.
Where you find the new docs right now
Documentation following the new system is being built and published daily by our Jenkins CI (currently by the documenteer job attached to the release pipelines).
You can find the latest daily release at https://pipelines.lsst.io/v/daily.
The latest weekly release is at https://pipelines.lsst.io/v/weekly.
Also, you can find documentation for specific daily and weekly release tags. For example: https://pipelines.lsst.io/v/w_2018_29.
Switching over to Jenkins-based builds
The next step is to entirely switch the build for https://pipelines.lsst.io from Travis CI to Jenkins CI. This change likely won’t become visible on the homepage until the next major release.
As a side effect, that means changes to the master branch of the pipelines_lsst_io repository aren’t automatically built and published to the homepage of https://pipelines.lsst.io — only a new major lsst_distrib release will update the https://pipelines.lsst.io homepage.
We (or at least, I) don’t have a clear plan of how we’ll introduce patches to the documentation of published releases. I’ll be talking to the release manager to see how we can introduce such a process.