Pipelines package docs hack day instructions

Data Management
#1

Update #2: As of 2018-12-17, these instructions have been superseded by the Developer Guide, which takes precedence. (Added by @swinbank)

Update: be sure to read the later instructions for how to continue doing numpydoc work after this hack day.

These are some rough notes for those working on Pipelines package documentation at the JTM 2018 hack day.

The goal for the hack day is to get packages configured with the new doc/ directory infrastructure, and begin to migrate docstrings over as well.

The coordination page is https://confluence.lsstcorp.org/pages/viewpage.action?pageId=73573930

For the hack day, we won’t be using JIRA tickets to track earned value. We’ll use user branches, like:

u/username/sphinx-docs

To claim a package to work on, add your name to the Notes column on the Confluence page.

The goal is not to get a package fully converted to Numpydoc today. As soon as you have a Sphinx-buildable doc/ directory (see below), feel free to merge your work. Lets not leave too many branches open after today.

Set up

In your set up stack environment, install documenteer:

pip install documenteer[pipelines]

(run with pip install --user ... if you’re on lsst-dev01).

This package brings in documentation dependencies, and includes our own Sphinx customizations.

Install the new doc/ infrastructure

The first step is to install some boilerplate files in the package’s doc/ directory. Refer to these:

The files you need to add are:

  • .gitignore
  • conf.py
  • manifest.yaml
  • index.rst
  • package_name/index.rst (package_name is the EUPS package name)
  • lsst.module.name/index.rst (lsst.module.name is the namespace of the Python package. For example, lsst.pipe.base, or lsst.afw.table)

All of these files, except for the .gitignore need to be customized.

conf.py

Set your package’s name and import the version module.

index.rst

Set the package title and set the paths in the toctree to the index.rst files in the lsst.module.name and package_name directories.

index.rst in the module and package subdirectories

Customize the titles of these pages and the anchor links.

Fill in the introductory paragraph you see from the template. Don’t worry about making it completely awesome.

For the package_name/index.rst, it shows a place to add the JIRA component. It’s hard to look up the JIRA component at the moment, so just put the name of the component there in plain text, like this:

JIRA component
    ``ap_association``

We’ll clean this up later.

manifest.yaml

In manifest.yaml, you’ll set the EUPS package name and the Python package name (or names, for situations like afw).

Here’s an example manifest.yaml from pipe_base: https://github.com/lsst/pipe_base/blob/master/doc/manifest.yaml

Build the package documentation locally

Once the boilerplate is added, you can try to build the package’s documentation locally.

First, make sure you’ve setup the package and run scons as usual. Since the package is being imported, any pybind11 wrappers need to be built first.

Then from the doc/ directory:

sphinx-build . _build/html

The root of the built HTML is _build/html/index.html.

Note that this is a single-package build. This is useful for building a package’s docs without building the entire pipelines.lsst.io site. Of course, at some point, there may be broken cross-package hyperlinks, but this is useful as a preview tool.

Beginning the Numpydoc conversion, one module at a time

You’ll find it easiest if you migrate docstrings one module (Python file) at a time, and rebuild the Sphinx site frequently to test for syntax issues.

Guidelines for writing Numpydoc docstrings are at https://developer.lsst.io/python/numpydoc.html

Once you think you’ve got a module documented, go to the doc/lsst.module.name/index.rst file and add a .. automodapi directive to the “Python API reference” section.

For example, if the namespace of the module is lsst.module.name.exampleA, the directive should be:

.. automodapi:: lsst.module.name.exampleA

For each module you convert, add an additional automodapi directive:

.. automodapi:: lsst.module.name.exampleA

.. automodapi:: lsst.module.name.exampleB

Eventually, once all modules are converted, you should consolidate all the individual automodapi statements to point to just the public module namespace of the package. For example, if the __init__.py at lsst.module.name imports exampleA and exampleB, then their public-facing namespace is lsst.module.name. The consolidated automodapi directive is:

.. automodapi:: lsst.module.name

Integrated builds

It’s not necessary for the initial package hacks, but if you’re curious, you can preview your package within the integrated pipelines.lsst.io site.

This involves using the tickets/DM-11216 branch of the pipelines_lsst_io repo. If you’d like you can create a temporary branch from tickets/DM-11216, but keep in mind that tickets/DM-11216 itself may be rebased at some point.

tickets/DM-11216 will be merged to master once the new-style documentation becomes the default and is fully supported with a Jenkins workflow.

Building the integrated pipelines documentation

These are the steps to build the integrated docs at the moment:

git clone https://github.com/lsst/pipelines_lsst_io
cd pipelines_lsst_io
git checkout tickets/DM-11216
setup -r .
build-stack-docs -d .

Adding your package

To test your package’s documentation in the integrated build, you’ll need to make the following changes to the pipelines_lsst_io repo:

  1. Add your package to ups/pipelines_lsst_io.table
  2. Add the lsst.module.name/index of your module to the “Python modules” toctree in the pipelines_lsst_io index.rst.
  3. Add the package_name/index.rst of the EUPS package to the “Packages” toctree in pipelines_lsst_io

Then try to rebuild the documentation with the build-stack-docs command.

I (@jsick ) will take care of adding packages to the DM-11216 branch as their upgrades land on master.

Using Python's venv with newinstall's Anaconda
Package documentation hack session at JTM 2018
#2

When you’re using automodapi, you’ll find that it will attempt to document all the APIs in your module, including those from imports of other packages.

What you need to do is use __all__ to curate the namespace of your modules to just the public APIs (functions, classes, module-level variables) that you are creating in that module.

#3

Another thing is that you might need to clean up the cached files that Sphinx/automodapi creates. From the doc/ directory,

rm -R py-api

Do this if the Sphinx build shows a lot of missing links while you experimenting with Numpydoc.

#4

Don’t worry about the warnings for missing _static directories. That can get cleaned up later in tooling.

#5

Merge advice:

Try to merge once your package is buildable with Sphinx (that sphinx-build command). In other words, you can do this as soon as you have the doc/ boilerplate in place. When you do this, mark off the “doc/ status” column on the Confluence page.

If you’re partway through Numpydoc conversion, you can still merge if the automodapi directive(s) are only pointing to the modules that have been successfully converted.

In principle, you should be able to do this merge without Jenkins, though if you’ve changed the docstrings and broke a PEP8 rule, flake8 may mark it as a failure. If you’re being cautious, a Jenkins run is always a good idea.

cc @mrawls

#6

For posterity, we had two recommended commits as part of the hack day:

  1. Update doc/ directory for sphinx - once you’ve got the relevant things copied from templates/example/doc and modified for the package.
  2. Cleanup numpydoc - once you’ve got numpydoc cleaned up enough that the sphinx-build command works.

Those would be the commits that go into your userbranch and once merged are enough to put the first check box on the confluence page.

#7

If there is still numpydoc work left uncommitted after the hack day, here’s what I recommend doing:

  1. Move any commits that update docstrings and __all__ attributes to a ticket branch (create a documentation ticket)

  2. Comment out the automodapi directive in the module’s index.rst. Put two full stops at the start of a line to comment it out:

    .. .. automodapi:: lsst.module.name

  3. Merge the doc/ directory work. You shouldn’t need to do a review or Jenkins run for this specific merge.

  4. Continue the docstring / __all__ work in the ticket branch. We recommend that you do the full ticket review process and Jenkins CI run for this work.

1 Like
#8