Congratulations on the DP2 release. I’m pulling a local copy of the DP2 DIA catalogs (DiaObject, DiaSource, and ForcedSourceOnDiaObject, about 25 billion rows in total) to run large-scale analyses on DiaObject light curves.
The setup: async TAP at data.lsst.cloud, staying within the documented quotas, with at most 10 concurrent queries, chunked by tract and by diaObjectId windows, with row-count verification per chunk. Some measurements from today that may be useful to the team:
Under current load, each chunk query takes 15–27 minutes to execute server-side, roughly independent of result size over the range probed (0.5–2 GB per result).
Queries whose results exceed roughly 4–5 GB reliably fail with Query failed in backend; below that they succeed. Can you confirm the actual result-size limit, so chunks can be sized against it rather than discovering it empirically?
RESPONSEFORMAT appears to be ignored on async jobs; results are always BINARY2 VOTable regardless of the requested format.
This workload sustains under 1% utilization of a 10 Gbit connection (about 9 MB/s averaged, arriving in ~1 Gbit bursts between long idle scans). The same machine routinely sustains ~8 Gbit/s from IPAC when bulk-downloading ZTF images, so the constraint is not network capacity or tuning on the client end: it’s qserv scan throughput under the concurrent-query quota.
Three questions beyond the result-size one above:
Is there a recommended path for full-catalog bulk copies that should be used instead of TAP?
Would a temporary concurrency allowance for a pull like this be possible? Scheduling it off-peak is no problem.
Is there a timeline for DP2 HATS/lsdb availability? The lsdb pages on dp2.lsst.io currently return 404. If HATS is imminent, waiting for it and sparing the TAP service is the obvious choice.
Thanks very much!
