AVRO schema wrangling

[roy (Roy Williams)]

The Lasair project is starting to be serious about AVRO and its schema.

• Where can I find the latest schema (.avsc) for the alert packets?
• How does the schema registry work?
• How can the community brokers find out that a new schema version is coming?
• Will the production schema have the “doc” attribute that describes each attribute in human language, so we can pass it on to our users?

[DouglasLTucker]

Hi, Roy! I am the forum watcher this week, but I admit I have no background on this particular topic. I see that @ebellm answered you when you asked similar AVRO-based questions back in January; so I’ve tagged him. I will also check around and get back to you a bit later today. Thanks!
Best regards,
Douglas

[DouglasLTucker]

Roy, I asked on the #desc-alerts-topical-team Slack channel, and @kessler pointed to this link: link . (Many thanks, Rick!). It looks like this may answer your first two questions, which is a good start. I will see if anyone can answer your other two questions.
Thanks!
Best regards,
Douglas

[DouglasLTucker]

Additional: It looks like @ebellm 's message from January may answer your 3rd question:

Schema Questions

Hi Roy, DMTN-093 contains our broad plans for schema management. To your questions in detail: We are not committing to never making breaking changes to the alert schemas, although obviously we will aim to maintain backwards compatibility insofar as is possible, particularly once the formal survey has begun. From a purely Avro perspective, changing the underlying computations without changing the alert fields wouldn’t require a version bump since the same schema would read both packets. …

Also, @rknop notes in the #desc-alerts-topical-team Slack channel:
Note that those [schemas in link] are the schema for ELAsTiCC, which are not the same as the schema for LSST itself. ELAsTiCC is not using a schema registry.

Re: knowing when there are new versions, specifically with regard to ELAsTiCC, the best thing to do is to be in the #elasticc-comms channel. We will also keep the latest schema in that github archive Rick linked, which is also linked from here: The DESC ELAsTiCC Challenge"

Again, all of that is ELAsTiCC specific. I’m not sure if Roy was asking about alerts for ELAsTiCC, or for LSST in general.

[ebellm (Еrіс Веⅼⅼⅿ)]

Hi @roy:

• Latest schemas are here
• see DMTN-210: Implementation of the LSST Alert Distribution System
• TBD, but I expect we will announce it brokers by email and Community. On the other hand the whole value of the schema registry is that if you get an unexpected schema ID you can query for it.
• yes. In the meantime you can see the relevant docstrings in the schema browser

[roy (Roy Williams)]

Thank you Douglas and Eric!

[gnarayan (Gautham Narayan)]

Re. a schema registry, I’d suggest it’d be useful for the project to coordinate with the scimma.org team and GCN/TACH so there is one solution with community buy-in, in the same way AVRO itself was adopted. Particularly for projects where the schema itself is unlikely to evolve rapidly, a solution like Confluent seems overly complicated and potentially expensive for smaller projects.

[ebellm (Еrіс Веⅼⅼⅿ)]

Hi @gnarayan Always happy to talk. Those projects don’t have the bandwidth constraints that Rubin does and have more heterogeneous messages: they almost certainly should just embed their schemas in their messages. I agree that the schema registry adds complexity, maybe unnecessarily, but it doesn’t preclude other means of disseminating schemas. The schema registry is part of the open-source portion of the Confluent platform, though, and in any case it’s just a REST API–there’s no cost for clients.