Dear colleagues
So far, several brokers are developed and their functionalities are presented many times. However, is there some performance considerations about them?
In RCW 2024, we are informed that “during Operations Rehearsal 4 (June 2024) Rubin Observatory processed simulated ComCam images sent from the summit and sent live alerts to brokers” (Eric Bellm’s talk). All brokers operated at the same time. I would be grateful if I could find some report about this test?
Thank you in advance.
What is performance? Surely you don’t just mean peak rate of ingestion? Might I suggest that what would really be useful is a collected overview of what each broker offers, its worldview, how it can address a given set of science questions. Suppose the scientist wants fast blue transients? A feed of possible/confirmed SNIa? Alerts within 50 Mpc? Repeated alerts from the same place in the sky? Possible AGN lensing? We need to know how each broker answers the science questions.
Yes, we did exactly this! Now, we would like to add anything quantitative / measurable / comparable / … if possible. Then, I remembered the talk from RCW where this test was mentioned, where all brokers operated at the same time, so I’m interested are there some results from this.
Hello @j.aleksic ,
The test was not intended to demonstrate anything in terms of performance, except perhaps that the Rubin analysis chain works from end to end over multiple nights.
Regarding performance, it is a common mistake to try to compare brokers. The seven brokers of LSST are not seven different implementations for a unique scientific purpose, but rather seven different implementations for multiple (non-overlapping) scientific purposes. The questions brokers try to answer depend on their communities of users – each having its own definition of what “performance” means (e.g. execution time, classification accuracy, classification purity, richness of contextual information, level of interoperablity, etc. with various thresholds). To compare two brokers, they would need to be asked the exact same scientific question to all brokers – but I doubt this is very interesting to anyone. If the use case is basic, all brokers will perform similarly. If the use-case is complicated, it is wiser to invest energy and time in working with one team at a time.
Many thanks! That clarified what I was interested in.