I remotely attended the Fall 2017 Astropy coordination meeting (held at the Center for Computational Astrophysics, Flatiron Institute, New York) the week of September 18. These are some notes on things I saw, discussed, and thought would be relevant to the LSST community.
The annual meeting is a chance for the astropy coordination committee to guide the future direction of astropy, work together on high-level aspects of the project including the project paper, new APEs (Astropy Proposal for Enhancement), and project communication, and also to bring together the rather distributed project members for a week of hacking and closing of issues.
The agenda and minutes from the meeting are available in this Google doc. Breakout planning was done using the github projects system; it was interesting to watch the latter used as a way to plan hack sessions and breakouts in real time.
My main reason for remotely attending was to try to finalize the NDData APE and catch up with some of the astropy folks on WCS-related questions.
It was interesting to listen to the various reports from different affiliate packages: astropy really does have a very broad user base. One of their open questions is exactly how to define the boundary between astropy-core and astropy-affiliated, and what to do when widely used affiliated packages lose their primary maintainer. These are problems that all open source projects have to deal with, but astronomy projects can require so much specialized domain knowledge that it can be very difficult for an experienced developer without that knowledge to take over a package.
Some samples from the week
Removing python2 support
See github issue #6556. This was mostly done as part of a day+ hack session lead by Brigita Sipocz. See also #6655 and #6234.
LSST will be going through this process in Spring 2018, and we may want to reference astropy’s work when planning that.
auto-closing stale pull requests
Astropy currently has around 90 open pull requests dating back to 2013. As a project run mostly by volunteers and with a minimal budget, they have decided to be somewhat aggressive about PRs that have had no human interact with them for a long time. They’ve implemented the astropy-bot to post a “Close?” warning and tag on PRs that haven’t had human interaction for at least 6 months, and subsequently auto-close them 1 month later. This may be extended to issues in the future, but it was felt that open issues often reflect bugs that are still live in the wild. LSST is a different project, with top-down management directives, but we may want to implement something like this for the Level 3 system in the future, as we currently do for our own RFCs (though that involves manual intervention by @timj).
More broadly, astropy has had good success with the astropy-bot to do things like remind users to include their changes/milestones in the changelog. The bot functions as an astropy-specific pseudo-CI system that monitors changes to pull requests.
LSST NDData APE
The NDData APE was started last year after the astropy/LSST summit at UW (March 2016). The goal was to unify the API of LSST’s MaskedImage and Astropy’s NDData systems for storing images+uncertainty+mask planes+metadata to allow future functionality sharing. The APE has proceeded in fits and starts, with significant progress usually happening after various in-person and remote discussions between the participants.
In this case, I talked with Matt Craig, James Turner, and a few others about our remaining differences, occasionally jumping on to LSST’s #dm-astropy slack channel for clarification of certain points.
Summary: Astropy agreed to add an origin object that will behave like our xy0, and a bit_plane that will behave like our mask, though keeping their own mask object that behaves like an ndarray mask (True means do not consider this pixel) for backwards compatibility. We believe that those were the final remaining points of contention, and astropy will have to decide how to adjust their current API (which Matt noted was probably overly complicated currently).
Hopefully we’ll be able to close out this APE soon and finally check off one of the significant items from the March 2016 summit.
generic pixel<->world WCS API
There was a discussion of an in-preparation APE about creating an API for performing pixel<->world transformations that could be shared by some of the astropy affiliates currently under development (e.g. gwcs and specutils). Some examples of world include RA/Dec on sky, wavelength, time, or even sky+wavelength for a spectral data cube. Note that I’m intentionally not linking here, as it is still under active internal development: I’ll make sure to share it with LSST on #dm-astropy slack once it’s posted.
Note that this is a much simpler interface than our afw.image.wcs. Its goal is to solve an immediate problem of some of the astropy affiliates in that astropy.wcs is not very general (being built on wcslib+SIP), and those affiliate packages need a generic API to do conversions between pixels and their relevant coordinate (sky, spectrum, etc.). Our initial look at the API suggested it wouldn’t prevent a more complicated system in the future (both GWCS and LSST’s AST-based system have many more intermediate layers). It’ll be nice to see the final APE and decide whether LSST can provide an interface layer for it.
As part of this discussion, I had Tomas Robitaille join LSST’s #dm-astropy slack channel to ask some questions to people there who had more knowledge of the IVOA’s standards documents. The process for getting a non-LSST member access to a slack channel is straightforward (just ask on #help-slack), and can be helpful to cut out the middle-man in a conversation.