2025-07-26 RSP @ data.lsst.cloud: LSDB, rate limits, RSP @ RCW

News Data Services
,
#1

:new: LSDB now available in Notebook environment

LSDB enables cross-matching on a large scale, running user-defined functions in bulk, and simplifies working with time-domain data. Check out the docs here: LSDB DP1 docs

LSDB is an open source community contribution to the RSP. If you are interested in contributing software or data to the RSP, read up on the process in technote RTN-105

:traffic_light: Rate limits active

As we’re coming up to the first century month of Data Preview 1, we have started to get a handle on your natural user patterns. This means we can now put in some limits to ensure fair resource and system availability without inconveniencing the vast majority ( > 99% in most cases) of users.

Things to know:

  • Limits can change at any time (in either direction) globally for all users, and also for specific users if we need to temporarily slow down someone whose usage is impacting a service in unexpected ways
  • We have set these limits to be well beyond typical usage and flirting with them is a bad idea. For example, if you exhaust your per-user concurrent query limit with an external pyvo script, you can lock yourself out of RSP services that also do queries for your behalf (like the portal). If you get a rate limit warning, back off your subsequent usage to below that limit.

The API endpoint listing your current limits is here: https://data.lsst.cloud/auth/api/v1/user-info - here’s what you are looking for:

  "quota": {
    "api": {
      "hips": 70,
      "sia": 70,
      "tap": 200,
      "vo-cutouts": 35
    },
    "notebook": {
      "cpu": 4,
      "memory": 16,
      "spawn": true
    },
    "tap": {
      "qserv": {
        "concurrent": 12
      }

The API values are per minute, and yes, we’ll roll out a pretty view of this information and improve the documentation around this in the near future.

Again, exhausting these limits will result in a degraded experience, so drive within the speed limit.

:woman_vampire: They Live In The Shadows (except during RCW)

For those of you heading to the Rubin Community Workshop in Tucson this week, there are credible rumors that some Data Services staff will be dragged kicking and screaming away from their laptops to interact with attendees without the benefit of a keyboard. You can marvel at this rare sight and ask your RSP questions at our Roadmap session.

2 Likes