Alert production is sensitive to how events are specified (VOEvent standard), authored and publishing streams (VOEvent Protocol). These are all pieces that sit outside the algorithmic effort specific to the Alert Production team. As such, it’s possible that they could be lost. I’ve been assuming AP would take responsibility of all three of these areas and that the Process Control team would provide an environment for the authoring and publishing to execute.
To be explicit:
AP will push forward any changes to the VOEvent standard necessary for LSST alert production to be successful
AP will provide an engine for alert authoring given a stream of alert records
AP will provide a system for publishing VOEvents to specified endpoints
Process control will collaborate with AP to design an execution environment that will enable the authoring and distribution system.
I hope this is in line with expectations by others, but please let me know if they are not: @ktl, @dpetravick, @gpdf probably have the most to say.
@gpdf Yes. I expect we will have some sort of bulk transfer system. This may end up just being a tarball of VOEvents per visit. The number of community brokers is expected to be small enough that we could potentially negotiate this mechanism on a broker by broker basis.
There’s quite a lot of identical metadata among all the O(10K) VOEvents from a single visit, right? I have been imagining that it could be useful to factor all that out into a single “VOEventGroup” header and then have only the per-alert data for each of the 10K.
More work to develop such a thing than the tarball approach, of course… it would require the existence of a library or libraries that recipients of the bulk stream could use to expand them out upon receipt. Maybe the answer depends a bit on whether the compacted form might also be useful internally to large brokers, or whether it is neither useful nor necessary in a world with sufficiently cheap bandwidth.
The IVOA has long kicked about the idea of a VOEventContainer, which fulfils broadly the same role as Gregory’s VOEventGroup. This has never really progressed beyond the brainstorming stage, but likely means that any concrete proposal for such a format from LSST would be well received.
I looked for the history of that term, and found primarily references to it in summary talks and papers of yours. Are there other proponents out there we should be consulting, or motivations different from ours to take into account?
At IVOA meetings going back several years, the pundits have sat around and nodded wisely at the idea of a VOEventContainer, but there have never been resources to take that further: I’m not aware of anybody writing down a concrete plan. Mike Fitzpatrick has certainly given this more thought than most, so he’d be an obvious person to consult for outside input. In general, though, I’d suggest putting together a skeletal proposal aimed at meeting LSST needs and running it by the IVOA TDIG mailing list for comments.
As part of the Data Access Centre for UK:LSST our department at ROE is looking at providing a VOEvent broker for the LSST events. As I understand it, the proposal is that we would receive a full event stream from LSST and then provide a filter and broker service available to the wider astronomy community. Does this fit with what you are thinking about the way VOEvent streams would be distributed from LSST ?
A number of our colleagues in the UK:LSST consortium are looking at transient events using the LSST VOEvent streams. In particular Stephen Smartt and Ken Smith at Queen’s University Belfast have experience working with the transient event data from the Pan-STARRS project and I believe they would be interested in contributing to the next generation of VOEvent specifications.