Buckle up, this is going to be a long one - and you’re going to want to read it.
Early DP2 now live at data.lsst.cloud
The Early DP2 release is now available on the Rubin Science Plaform at data.lsst.cloud through all services (portal, notebook, API).
This publication includes:
- The complete DP2 campaign catalogs
- The complete DP2 campaign set of deep_coadds, our most advanced image product
- Key auxilliary products such as survey property maps, object_scarlet_models, skyMaps, and visit summaries.
The remaining DP2 image products will come in the full DP2 publication, as soon as we have reached the capacity to serve that kind of data volume reliably and performantly. RTN-11 remains the canonical source of our best schedule estimate for that and other upcoming releases.
For more on DP2 see dp2.lsst.io. If you’re a first time Rubin Science Platform user - or even need a refresher - check out rsp.lsst.io. Questions? You can ask them right here under the Support category, don’t be shy.
New image file format (don’t panic)
The Data Previews are not just sneak peeks for you, they are learning experiences for us, and one of the things we learned was that we were… let’s say somewhat unsatisfied with our data serialization. For DP2 and going forward, image data products are written in a new file format, with brand-new classes to represent them in Python.
The new format is still FITS-based [1] but does better at being self-documenting, better at handling versioning changes, and opens the door to more file types in the future, such as ASDF or HDF5 [2].
The new Python types are way easier to understand, especially if you’re comfortable with Astropy. And, the new classes are in a pip-installable package (lsst.images) so you can read the images from your code without needing a kitchen-sink LSST Science Pipelines installation.
For much more on this, read up on What’s new with DP2 images.
New recommended - v30.0.10
When you next use the notebook service at data.lsst.cloud you will find a new recommended image based on the v30.0.10 version of the pipelines. This is the best choice for working with DP2 data, or trying out the many tutorials available.
If you need (much) more details, consult the pipelines release notes.
Ugh bugs
A consequence of the hot-off-the-presses new file format is that there are a couple of bugs floating around, the most pesky of which is an issue with a third-party library that is used by some java-based viewers. A symptom of this is that our Rubin Science Platform portal is having trouble rendering image cutouts (images are fine). We have upstreamed a fix, that developer is actively working on a release, and as soon as we have it we will update our portal to resolve the problem.
We don’t like putting out releases with known bugs like that[3] but it seemed a shame to hold back DP2 for just that so, fix incoming in a week or two. And sorry.
Portion control
(Worried everybody would skip over a section called Quotas. Wait, come back!)
We apply usage limits in the Rubin Science Platform so that we can reliably, fairly and affordably provide services to our community. These limits can change at any time - generally upwards with time, as we scale the platform, but if something crazy is going on, we can also briefly dial one or more service limits downwards, for a specific user or all users.
If this all sounds a bit opaque, worry not. You can now find values for all quotas and limits applied to your account by visiting the new Quotas section of your RSP settings (found under the menu with your username on at data.lsst.cloud). The values you see there are correct for the time you visit the page, since they are drawn from the live system. For more on what these values mean, check the docs.
Oh and we don’t judge you for triggering those limits, but it’s inefficient for you as well as for us if that happens. It is better if you regulate your API usage to stay below them.
More (memory) for less (time)
Speaking of quotas and limits, we frequently pore over our metrics to find good trade-offs on the basis of your usage patterns. For example, recently we doubled the memory available for notebooks to 32GB since we noticed a lot of out of memory session terminations.
To compensate, we lowered the maximum running time for notebooks to 5 days, though sessions can be terminated early for a bunch of reasons, so always write your notebooks in a way that you won’t lose your work if that happens.
New alert retrieval by ID service
Services built on our prompt products database (PPDB) will be our next major release. In the meanwhile, we have a new service that, provided you know one of our Alert IDs (by subscribing to your favorite alerts broker), you can retrieve the complete alert packet directly from the RSP.
Technical documentation for the new service can be found at rsp.lsst.io, and there is also a tutorial demonstrating example usage.
For our best estimates for the release of PPDB services (or any other data releases), RTN-011 is the place to check.
Data Services @ RCW
We’re a bit thinner on the ground at RCW this year (something about being locked up in a basement shipping EDP2) but if you see any sleep-deprived geeks squinting in the sunlight, come and say hi - it’s probably us, and if not they probably need a fistbump anyway.