Currently, the Rubin Science Platform (RSP) provides a default storage quota of 35 GB per user. The Resource Allocation Committee (RAC) will review requests for resources beyond these default allocations. I would like to understand the policy and procedure for requesting larger allocations.
Is the Resource Allocation Committee (RAC) already operational, and what is the expected timeline for reviewing resource requests? What is the procedure for requesting increased storage beyond the default quota on the RSP?
Up to what size can users request, particularly for collaborative projects? For example, can a science collaboration like the strong lensing community request a few TB to store intermediate products such as time-series cutouts? The intention is to store this intermediate data close to where the processing and analysis are performed, in order to minimize data access latency.
In addition to storage, can users request additional compute resources? Is compute resource pooling among users feasible and advisable for collaborations running large-scale processing?
Finally, is there any possibility of GPU availability in the future to support deep learning workflows? I am aware that GPUs were not part of the initial plan, but I would appreciate any updates on this front.
Hi @deltasata , thanks for these questions, I think I can answer them all for you.
No, the RAC is not yet stood up or operational. There is a Rubin Tech Note (RTN) that outlines the vision for the future RAC: rtn-084.lsst.io. The procedures and timelines would be provided in the future calls for proposals to the RAC.
Exactly what resources will be available to be allocated, and when, are still being determined and are likely to evolve with time. They would also be provided in the future calls for proposals to the RAC. But yes, storage of intermediate products sounds like a reasonable use-case for the RAC.
Yes, in the future the batch compute resources will be allocated by the RAC, and also yes the vision includes enabling teams to propose and collaborate.
With respect to GPUs, here I’ll quote the RSP Roadmap: “…we are investigating ways to competitively provide access to GPU and/or other resources friendly to machine learning.”
I think that about covers it so I’m going to mark this reply post as the solution, but please don’t hesitate to follow-up or open a new Support topic any time.
I think our feeling on this sort of thing is that if you are needing to generate multiple TB of intermediate products from a data release it’s much more efficient to handle this as a request to the RAC to do batch processing at the USDF/SLAC and then store all the outputs on SLAC storage, rather than trying to pull over data into the cloud storage area.