Schema Questions

Science Alerts & Brokers
#1

I have a couple of questions about I would ask about schema changes.

What I recall from the ZTF experience is that a new real-bogus system was introduced (attribute drb), and Lasair needed to be changed to store it, and also our documentation changed to show users the new way to build queries. So there was a machine part of the adjustment (ALTER TABLE ADD COLUMN), as well as the human part. However, I note that the old way to do real-bogus (rb/nbad/fwhm/elong) still works, so nothing fails, so hooray!

– Will this backward compatibility still hold for LSST? That new attributes will be added, but old ones never removed? How about versioning the way attributes are computed, will that be part of the schema changes?

– What will actually happen when the schema changes? I assume brokers will be informed well in advance, together with a “switch day”, so we can prepare for it. I don’t remember getting such messages from ZTF, but maybe I just missed them.

– Next is human-readable definitions of the attributes. The ZTF schema has a “Contents” column for this, and Lasair has a similar structure that propagates all over. I see this information for LSST in the DPDD, but it would be great to have this machine-readable and versioned.

Thank you as always
Roy

#2

Hi Roy,

DMTN-093 contains our broad plans for schema management. To your questions in detail:

  • We are not committing to never making breaking changes to the alert schemas, although obviously we will aim to maintain backwards compatibility insofar as is possible, particularly once the formal survey has begun.

  • From a purely Avro perspective, changing the underlying computations without changing the alert fields wouldn’t require a version bump since the same schema would read both packets. I could see the value to brokers in being able to distinguish the two, however. I think we’d address this when or if it becomes an issue.

  • We’ll certainly provide plenty of advanced notice before a schema change, likely through announcements here as well as directly to the broker teams. Recall that Rubin is using a Schema Registry so you can programmatically retrieve schemas at any time.

  • I have a neglected ticket to add the attribute definitions.

Eric

AVRO schema wrangling