I’m currently working on uploading cosmoDC2 data files obtained from Globus into the RSP. I’ve been first transferring files into my computer and then uploading them into the RSP, however I was wondering if there was a way to upload the files directly into the RSP.
To transfer files from Globus, I need to specify a path, which I have been routing to my desktop. The Globus website says that files can be transferred into “endpoints” on shared servers if the endpoint has already been set up. Does anyone know if such an endpoint exists, or of any other way to directly upload these files into the RSP?
Hi, there is currently no file upload service to the RSP. Note that while quotas are not currently enforced during Data Preview 0, individual user allocation is aproximately 1TB/user, so non-bulkload transfer methods (wget etc) work reasonably well in that regime. If you have data that is of general use to the RSP community and you have the data rights to make it available to all users, please email me and we can discuss how to get it to a public deployment.
[This reply assumes you are talking about the flagship RSP deployment on data.lsst.cloud - if you are working on a deployment of the RSP by an IDAC and this is a usecase that IDAC wants to support I can advise your provider as to what would in developing such an endpoint]
Thank you for your response! I definitely should not exceed 1 TB with the data files as I’m only uploading a few at a time and extracting the much smaller amount of information that I need. Thanks again!