Roll-out of continuously deployed design documents

Following up on Plan for Planning:

I’m now just about ready to begin handing over documents to authors.

I’ve used LDM-152 as a prototype. You can see the

The visual design is forked from the default readthedocs.org Sphinx theme and was gently tweaked to suit singe-page documents like ours. I can do some additional modification to colours, fonts, etc. to better suit our needs. More dramatic re-designing will probably require implementing our own theme from scratch. But this should definitely be sufficient for developing the design documents.

I’m thinking it may be good to put together some docs early next week to specify the relevant bits of reST syntax and other information needed to work with the Design Documents.

Now that I have the templates, content conversion and build process mostly developed for Word→reST design documents, I may have time next week to even roll out continuous web deployment of LaTeX-based design documents. I think I can use Travis CI, GitHub pages and tex4ht to quickly get something together.

1 Like

Awesome, @jsick .

For now, I’ve made the LDM-152 repository writable by the DM Team. Without per-branch permissions on GitHub, managing change control technically (as opposed to socially) may be tricky. Options:

  1. Go to a fork/PR model. But auto-rendering proposed changes would be difficult.
  2. Add a status check that includes TCT signoff for merges to master. This may be easier to do, as I’m sure that “review signoff” status checks must exist. (But does this feature of GitHub work for merges that people do from the command line, or only the “green button” in a PR?)
  3. Something else?

if you want merge control then I think this is the best approach. Can the Github Travis integration help with the rendering problem?

@timj Can you clarify on what the rendering problem is?

I am guessing that reviewers would like to see the words in ReadTheDocs form rather than Github PR diff form and I’m wondering if they’d like the changes highlighted in the web page view. All guess work though.

Oh sorry, I missed @ktl’s mention of rendering the diff on the webpage itself. While that would be über cool, I doubt I could deliver that next week

I really think that authors should make use of GitHub’s prose diffing capability to see changes in the reStructuredText itself. The reST itself is so readable that making the extra leap to rendering diffs on the website would be little bang for lots of bucks, in my opinion, at least.

No need to render the diffs themselves; I was thinking of just rendering the head of the branch/fork. In cases where there are extensive changes, that might be more readable. But don’t worry about that for now.

Another update:

Word → Conversions

Besides LDM-152 we’ve also got PRs out for LDM-129 and LDM-230:

I’m also working on @jbecla’s magnum opus LDM-135 but have gotten bogged down in reStructuredText formatting fixes.

I’m returning briefly to the Canadas September 24—28, so there will be some delay in my responses to code reviews for LDM-{152,129,230} and the LDM-135 conversion.

On LaTeX documents

And while I’m excited about supporting Travis CI +GitHub pages + tex4ht for LaTeX documents, it will have to go on the backburner while I get on with the Stack docs. Luckily that won’t impact editorial workflows for the LaTeX documents. Eventually having an LDM index page off of http://dm.lsst.org and/or the Stack docs where nicely-presented Design Documents are easily accessible in the browser is very much my goal.

Thanks bunches, @jsick! In a cursory scan I haven’t seen major issues with the LDM-153/129/230 conversions, but I’ll take a closer look and sign off on the reviews.

Of course, the CSS needs a lot of work (especially for tables and figures). The styles also need to reflect the LSST branding. But anyways, I’ll get to that at the same time as I’m designing the Stack Docs.

I’ve completed conversion work of @jbecla’s LDM-135. See https://github.com/lsst/LDM-135/tree/tickets/DM-3835

The instructions there (and all the other LDM documents I’ve converted) tell you how to build the document into HTML.

Unfortunately, readthedocs.org has recently started choking on my builds, see this GitHub issue. I think I can work-around this issue by releasing my Documenteer and lsst_dd_rtd_theme as properly released PyPI packages rather than installing from GitHub.

Once I’m confident in the readthedocs build I’ll formally put ticket DM-3835 into Review.

LDM-135 was an interesting (and expensive) challenge for converting to reST. I think I need to provide better support for bibliographies with the niceties that LaTeX/natbib provide. For now I’ve converted some of the LDM-135 references into inline links where it makes most sense, and used reST’s citations markup where an actual document is being cited. The formatting of these references, and their citation keys, isn’t perfect, though.

Thanks Jonathan! I am trying to follow the instructions, at the moment the requirements.txt is only in the branch, so one has to checkout tickets/DM-3835 first. Installing requirements gives me “fatal error yaml.h: No such file or directory”, and “couldn’t create /usr/local/lib/python.2.7/dist-packages/yaml: permission denied”. Regarding the later, if sudo it required, it should be mentioned. I’ll let you comment on these before digging any further.

Use a python for which you have permission to install packages. Are you using the system python? Either use a virtualenv or just install your own Python.

The instructions in the README work for me if I use my own Python (which happened to be Python 3).

I’m happy to, but I still think that should be mentioned in the instructions.

using virtualenv solves both problems, so the 1st one was related to permissions.

@jbecla It’s true that you may need to pip install with sudo if you’re not using a Python virtual environment. I’d recommend you always use a virtual environment (via virualenv/conda/venv) whenever doing anything in Python, especially installing someone’s random python package for testing :wink:

I realize that virtual environments aren’t part of DM’s culture, so I’ll write some boilerplate instructions I put in all my project READMEs.

I also realized that the instructions are assuming a build from master, but of course we’re still on a ticket branch at this point. Here are the build steps for this case, assuming you still need sudo:

git clone git@github.com:lsst/LDM-135.git
cd LDM-135
git fetch
git checkout origin/tickets/DM-3835
sudo pip install -r requirements.txt
make html

Also, since @timj tried it; I’m not supporting the LaTeX-based PDF builds of the Sphinx/reST docs at this point. My plan is to deliver a better Sphinx HTML theme and make that print-ready.

tl;dr don’t do make latex :slight_smile:

In 5.5.5, would it make sense to wrap the text inside cells instead of having scrollable table?

Also, I am sure people will be asking how to print it, or generate pdf

Both of those will be goals/outcomes of delivering and owning the HTML/CSS/JS stack for Design Documents. I’m using a mildly-hacked ReadTheDocs theme for now so that authors can at least have something to edit and continuously deploy to the web.

I made first full scan, it looks great! I pushed few minor fixes to your branch. Thanks a lot! :slight_smile:

1 Like