Hello Lasair Team,
I’m experiencing an issue with my Kafka stream connection to the Lasair alert broker. Although the Lasair dashboard shows active alert ingestion, my active streaming filters are not receiving any new alerts.
Here’s a summary of the situation:
-
Filter setup: I have both a tightly scoped filter (with classification and magnitude thresholds) and a completely loose filter (no WHERE conditions, selecting basic metadata like
objectId,jdmax,jdmin,gmag, etc.). -
Kafka topic setup: I’m using the topic name generated by the filter (e.g.,
lasair_1568BrightFastTransients) and connecting via the recommendedlasair_consumer()method. - Group ID behavior: Even when I create a new Kafka consumer group ID, which should allow replay from the Kafka cache (~7 days), I only receive a small number of alerts (e.g., 10). No new messages arrive after that, even over multiple hours.
-
Logs & observation: My logs show consistent Kafka polling with timeouts. No
msg.error()or parsing issues are occurring. Polling is active and functioning, but no data is coming through.
This seems inconsistent with the Lasair metrics, which indicate that alerts are flowing through the system or am i misinterpreting them.
Questions:
- Is there currently any known issue or delay in the Kafka streaming filter infrastructure?
- Is there additional configuration needed to ensure that filters are actively triggered and producing Kafka messages?
- Could there be a backpressure or throttling mechanism delaying the emission of Kafka alerts from newly created filters?
Please let me if any more info is needed.
Here is my existing codebase: Zwicky-Transient-Facility-Consumer/src/consumer.py at master · bitterfq/Zwicky-Transient-Facility-Consumer · GitHub
topics:
lasair_1568loosetestfilter (Filter: loose test filter)
lasair_1568BrightFastTransients(Filter: Bright Fast Transients)
Thanks,