What goes in a VOEvent
Executive summary:
The original intent was to capture any thoughts on what should go in a VOEvent that is sent out to the public. This was predicated on the assumption that all subscribers would see identical VOEvents. Since we expect to send ~10 million alerts per night, it is important to optimize the contents.
The conversation ended up being focused instead on the architecture of the event generation pipelines. @mjuric pointed out that the alert is not the VOEvent packet. Instead, we should think of the alert as the DIASource and all ancillary data. In this view, the filtering mechanism provided by LSST could be able to both cull alerts and format the resultant VOEvent. If this is the case, users will naturally only send on the minimum amount of data to get their science done. The question then becomes what else we can provide the filter to be useful (opposite of the previous question what can we avoid sending).
@gpdf pointed out that allowing all alerts to be packaged independently in many different ways makes it very difficult to find all VOEvents derived from a given alert. It seems like this concern could be alleviated by producing a “master VOEvent” at the beginning of the pipeline for each alert which would be signed by LSST and referenced by any child VOEvent.
Actions:
There were three major actions taken from the discussion.
Action 1: It would be very useful to make the full Object record associated with a DIASource available to the filtering mechanism. The current baseline is to just provide an ObjectId. This has a cost implication and requires a change to the DPDD. @connolly and @ivezic are asked to draft a proposal to send the the CCB. This will need input from @ktl.
Action 2: @KSK and @connolly will take the sketch of the architecture presented by @mjuric and incorporate a fleshed out version in LDM-151.
Action 3: @KSK and @connolly need to plan how to determine the optimal way of associating Objects with DIASources.
Full notes follow
Attendees:
- Tim Jenness @timj
- Simon Krughoff @KSK
- John Swinbank @jdswinbank
- Mario Juric @mjuric
- David Reiss @reiss
- David Ciardi @davidciardi
- K-T Lim @ktl
- Hsin-Fang Chiang @hsinfang
- Melissa Graham @MelissaGraham
- Andy Salnikov @andy_slac
- Veljko Vujcic [not at-mention]
- Xiuqin Wu @xiuqin
- Gregory Dubois-Felsmann @gpdf
- Darko Jevremovic @darko
Minutes:
Overview of baseline
- Several people pointed out that the way associated objects are defined needs to be planned
- Verify object records will be sent with the alert and update DPDD
- K-T: has a cost relative to association (need to preload additional data from Object table, may mean spilling to disk instead of doing entirely in-memory).
- Gregory: Could make alert db public
- Tim: ANTARES is thinking of making a public alert db.
- David C: What happens with alerts that get dropped? – There wasn’t an immediate answer to this because it depends strongly on what the architecture looks like.
Nominal architectural design
Level 1 processing sends DIASources, associations and images to one or more authoring systems which create VOEvents and then publish them to brokers.
Proposed architectural design
- Mario: If we put a message queue (or something) in the path before the filtering step, the filters could use all the data and only pass on what is needed in the published VOEvent.
- Gregory: But this design means that the events are not atomic. Everyone gets a different event. That’s a bit scary.
- Gregory: Will all DIASources end up as an alert?
- Mario: Unclear, but there will likely be a step where DIASources are filtered for obvious false detections.
- Gregory: It would be nice if we could save sources at say 5sigma (or something) but only alert on ones that are 5.5sigma.
- Mario: This is a possibility, but is deferred for further investigation by the AP team.
- John: Event packet has an identifier, so if we are going to refer to events by DIASource ids, there are ways to deal with this, but it makes things a little complicated.
- Gregory: Concern is that we cannot send an alert that we are signing (no master signed VOEvent).
- Mario: We could mitigate the concern by putting a master VOEvent author at the level of the message queue.
- Gregory: That would help because there would be a master VOEvent to reference. If alert is modified downstream, we will know that.
Overview of VOEvent standard
Some discussion on including postage stamps in VOEvents.
- John: No binary data in the standard
- Mario: You can include base 64 encoded data.
- John: Not sure what that buys you…
- Gregory: What does including the stamp by value buy you?
- Mario: Allowing people to write filters that depend on postage stamps.
- Gregory: Isn’t this just a latency question?
- Mario: I’m more worried about supporting many callbacks into a service. The tradeoff is really do we support a robust service for users to callback to, or do we just include the postage stamps?
- Gregory: What fraction of the users want the values and what fraction of the bandwidth is that?
- Tim: From ANTARES POV, postage stamp was going to just be forwarded. The postage stamp is stripped out first thing and only forward on the postage stamp in downstream events. Do brokers actually want the postage stamps?
- K-T: It is a more conservative design to send them by value
- Mario: There is a question about when the cutout server will be available.
- K-T: Immediately with no guarantee on latency.
Discussion of LSST needs and how they map on the standard
- K-T: Exposure metadata?
- Mario: We need to do that, but I don’t know how.
- John: The element supplies instrument specific information.
- Gregory: Could be transfer level not packet level. We know we are going to be packaging events at some level, so the common information can be distributed in one copy with each package.
- John: I think this does imply two different transport protocols. (Or even two different types of VOEvent, or VOEvent and VOEvent-in-transit as separate entities)
- Gregory: Exactly it would be a different thing all together.
- Simon: We should include event references to all children in the master Event.
Process for engaging with IVOA on modifying the standard to our needs.
- John: To what extent do we care if our events can be authenticated?
- Gregory: When I said signed, I meant really signed.
- John: There is no good way to sign events at the moment.
- Gregory: In XML this is hard because the attributes are not ordered.
- John: There are attempts to form canonical forms of an XML document and sign that. It is very complicated, but basically works.
- Simon: What would we use if we were doing VOEvent 3.0?
- Tim: Brian V. suggested protocol buffers (or capnproto).
- Andy S: could not comment at a time but I can add that that protobuf is not self-describing which makes it less useful in my view.
- Veljko: What about JSON?
- Tim: XML and JSON compress similarly, is my impression. We need to compress before we go over the wire (if text) anyway.
Side discussion
- Gregory: Worried about the transition from a push to pull is a major scope increase.
- Mario: This was a straw man. No need to interpret this as a change to a pull.
- Gregory: Where does funding come from to fund community broker? We need a simple broker to make sure we can send alerts on day one. We can take some pressure off the community by making a better broker, but maybe that’s bad.
- Gregory: Major worry about the provenance. We need a maximal event as a reference.
- John: Could be a minimal event.
- Gregory: but having a maximal event as the reference defines the reference schema.
- Gregory: There is a concern about having a DB in-line. Intermediate persistence could help remove the DB while still preserving latency and consistency.