New PredictedQuantumGraph class and file format

Data Management DM Notifications
#1

I’ve just merged (in time for d_2025_10_01) DM-51850, which adds a new lsst.pipe.base.quantum_graph.PredictedQuantumGraph class and corresponding file format, which will soon be fully replacing lsst.pipe.base.QuantumGraph. This switch should be invisible to almost all of our code; interfaces that took QuantumGraph now accept both, and interfaces that returned QuantumGraph generally have counterparts with new names that return the new PredictedQuantumGraph. The old class isn’t deprecated yet: this is the first step in a series of related changes, and I’ll do one RFC for the deprecations closer to the end.

But you will probably see at least marginally better performance by switching to the new file format, which you can opt in to from the command line by using the .qg suffix instead of .qgraph for your filenames. Performance for pre-exec-init (i.e. the BPS pipetaskInit job) should be dramatically better with the new format.

#2

I’ve now also merged (in time for w_2025_40) DM-52339, which makes .qg the default file extension for BPS submissions and updates BPS to use the new PredictedQuantumGraph class for generic workflow creation and clustering.

The upshot is that BPS will now only need to read a small fraction of the QG, which should take a few seconds instead of many tens of minutes; combined with performance improvements in writing QG files, I expect this to speed up BPS submit time for the very largest graphs by about an hour overall.

We do not yet expect significant improvements to BPS finalJob or pipetask report performance, but work is underway.