SPIE Abstract: Investigating interoperability of the LSST Data Management software stack with Astropy

I intend to submit the following abstract to the SPIE “Software and Cyberinfrastructure for Astronomy” track at next summer’s SPIE Astronomy conference. This will be based on work I will be doing February to April. Let me know if you would be interested in joining the effort (at the level of being a person with an opinion who can join in discussions: I know that explicit work is not scheduled for others in this cycle) and I’ll add you to the author list. Formal decisions on authors for the submitted paper can wait until April when the paper is written. Comments welcome on the abstract (submission deadline is a week away).

Investigating interoperability of the LSST Data Management software stack with Astropy

Jenness, Bosch, Owen, Parejko, Sick, Swinbank et al

The Large Synoptic Survey Telescope will be an 8.4m optical survey telescope sited in Chile and capable of imaging the entire sky twice a week. The data rate of approximately 15 TB per night and the requirements to both issue alerts on transient sources within 60 seconds of observing and also to create annual data releases means that automated data management systems and data processing pipelines are a key deliverable of the LSST construction project. The LSST data management software has been in development since 2004 and, like other software developed in that era such as CASA, is based on a C++ core with a Python control layer. The software consists of nearly quarter of a million lines of code covering the system from fundamental WCS and table libraries to pipeline environments and distributed process execution.

The Astropy project began in 2011 as an attempt to bring together disparate open source Python projects and build a core standard infrastructure that can be used by and built upon by the astronomy community. This project has been phenomenally successful in the years since it has begun and has grown to be the de facto standard for Python software in astronomy. Astropy brings with it considerable expectations from the community on how astronomy Python software should be developed and it is clear that by the time LSST is fully operational in the 2020s many of the prospective users of the LSST software stack will assume that Astropy software will be integrated.

In this paper we will describe the overlap between the LSST science pipeline software and Astropy software and investigate areas where the LSST software provides new functionality. We will also discuss the possibilities of re-engineering the LSST science pipeline software to build upon Astropy, including the option of contributing affiliated packages.

1 Like

This is a great idea. Maybe it is understood but you could mention in the abstract that Astropy is an open source community effort.

Thanks to everyone for their comments. Abstract has been submitted to the Software And Cyberinfrastructure for Astronomy IV conference. @jsick, @jbosch, @jdswinbank, @rowen and @migueldvb as co-authors. I can update the author list up until next Monday.

Do you have thoughts on a timeline for us to kickoff the project and divvy up the investigation?

I have opinions on this topic! Is there something else I should look at, or just the abstract above? I should be able to contribute some text eventually, too.

I’m allocated to other tasks until the end of January. That doesn’t mean that we can’t work out communication schemes and basic background stuff before then. HipChat room? Slack? Here?

We have to start with AFW and write the core classes down in a table and then work out what the astropy equivalents are.

Paper (minimum 6 SPIE pages) has to be written and submitted to pub board by end of April so that’s the main deadline. Summer 16 starts too early for any of this to have any effect, but some of the outcomes from this work could be used as input to a summer 16 replan at the end of May.

Great. I’ll add you to the abstract author list.

The abstract is the only concrete statement of work at the moment.

Not Slack, please. It is not yet an official communications mechanism and as such should not be used for anything serious.

Just a comment that this paper is being developed on Github.