- Summary: Lasair’s output Kafka from active filters now has options that add the lightcurve to each Kafka record. But there is a quota, so each filter must choose to have a lot of records with little data, or few records with a lot of data.
The Lasair Kafka streams have been extended and expanded. As you recall, every active filter generates either a Kafka stream or an email stream, and this article is about the Kafka stream.
Kafka from Lasair is in JSON format, and each record in the stream corresponds to a single alert from Rubin that has passed through your active filter. If, for example, the SELECT part of your filter says objects.diaObjectId, objects.decl, objects.ra, then the corresponding Kafka record might be like this, where Lasair has added by default the creation time of the record:
{
"diaObjectId": 169760235333878021,
"decl": -38.16173,
"ra": 221.87087,
"UTC": "2026-01-29 11:40:14",
}
Users were encouraged to follow up receipt of a Kafka record by using the Lasair API to get full information. But now, the expanded Kafka offering has two more choices when saving a filter in the Lasair web:
- kafka stream with lite lightcurve
- kafka stream with full alert
In each case there is an alert key in the JSON that contains diaSourcesList and diaForcedSources lists. For the “lite” option, there are five attributes for each diaSource:
-
psfFlux, psfFluxErr: the difference flux and error (nanoJansky), -
midpointMjdTai: time of the detection (MJD), -
band: string that can be one of u, g, r, i, z, y. -
reliability: Rubin’s measure of reliability of the difference detection, between 0 and 1, with 1 being most reliable.
The “full” alert option returns a large record, that includes the diaObject of over 80 attributes, and 100 attributes for each diaSource, of which there may be 100s.
However, these expanded options are much larger than the original thin Kafka stream, especially as the LSST progresses and the number of diaSources and diaForcedSources in the lightcurve gets large. Therefore, this new capability comes at a cost, and each filter has a quota.
Each new filter can produce up to 100 Mbyte per day, and users must decide on either a very selective filter with full alerts as the output, or a less selective filter with thinner output.
Full documentation about alert streams and Kafka is in the Lasair Documentation