Rubin Operations’ Response to the UC 2023A Report
Thank you to the Rubin Users Committee for the 23A report.
Words in italics are text from the UC’s report above.
Regular text are responses from Rubin Observatory staff.
A. Follow-up from the 2022B report.
(1) Additional simulated datasets. 2022B: We recommend that the project provide simulated data sets for other science communities. 2023A: The UC is pleased with the work toward incorporating simulated solar system observations DP0.3.
Rubin Ops thanks the Solar System Science Collaboration for making the DP0.3 simulation and sharing it with all Rubin data rights holders.
(2) Simulated alert packets. 2022B: We recommend the brokers or project provide a set of simulated alert packets based on DP0.2. 2023A: The UC encourages the project to continue to pursue this.
Rubin Data Management is currently working to generate simulated alert packets from DP0.2 that will be distributed to Rubin Community Brokers. We expect this to start well in advance of the First Photon milestone (2024-07-16). More details will be circulated in 2024.
(3) Validation reports. 2022B: We recommend the project provide a characterization and validation report for DP0.2. 2023A: The UC still thinks such a report would be very helpful. A report for DP1 may be too late to be of full value.
It is still the case that full verification and validation reports will be provided for all data releases starting with Data Preview 1 (DP1), and will not be provided for any simulated data products. DP1 is when the community will be able to start to prepare for LSST data and there will be sufficient time for feedback after DP1’s release.
B. Findings.
(1) Portal Aspect instructions and tutorials. The instructions and tutorials provided in the portal aspect are well structured and clear. Perhaps something that could be emphasized more in the instructions is the link between the portal and the notebook aspects. For example, what are the advantages (if any) of the portal aspect compared to the notebook aspect (and vice versa). If there are tutorials that show how to carry out the same identical tasks using the portal and the notebook, highlight these tutorials as a way to link the two aspects. If there are things that cannot be done easily in one of the aspects, it could also be useful to point that out explicitly in the instructions.
The Rubin Community Science Team (CST) thanks the UC for reviewing the Portal Aspect instructions and tutorials.
Regarding tutorials that carry out identical tasks, we have been thinking along the same track and for DP0.3, the introductory Portal and Notebook tutorials will cover the exact same steps. Provided this works out as well as we all suspect, we might rearrange some DP0.2 Portal/Notebook tutorials to be “matched” in the same way, and would certainly continue to do the same for DP1.
Regarding the advantages of one aspect over another and things that cannot be easily done in one aspect, a few words about this has been added right to the top of the RSP instructions in the DP0.2 documentation.
(2) The API Aspect. In addition it would be good to further emphasize the API aspect. Not a lot of time has been spent training DP0 delegates to use this; it can be a very effective way to interact with DP0 data products through software such as TopCat.
The Rubin CST agrees, and has work tickets in place with due dates at the end of September 2023 (end of FY23). The UC will be able to review the new tutorials in advance of their 23B fall meeting.
(3) User-installed software. The documentation related to user-installed software capabilities is good but as managing a working Conda environment is particularly tricky, especially if it needs to be extended with external packages, a more detailed set of instructions with examples on how to do this would be useful.
The Rubin CST agrees that the current FAQ on “How do I install packages in my user environment?” is insufficient, and has work in progress to update the advice for user environments by mid-August. There is also now a tutorial notebook available (Find DP0.2 tutorial #11, “Working with user packages”, in the tutorial-notebooks repo).
C. Requests for new/continued functionality.
(1) Code environments. An R environment in the RSP. This might involve supporting R notebooks and/or using the RPy2 Python-R bridge that allows efficient calls from Python numpy to R functions.
Rubin Observatory’s priority is to concentrate on providing a good user experience and support for one language, Python, across all Rubin services. Rubin observatory has looked at providing support in other languages and concluded we do not have the resources to do it. Any new language requires significant effort that is not available (e.g., building an R interface for the Butler, for the Engineering Facilities Database, etc.). Python is the most widely used language in astronomy and in addition, all Rubin/LSST software is written in Python (except for some C++ packages that all have Python user-facing interfaces). This is why Python is the chosen language.
(2) Image viewer. An improved interactive image viewer for the Portal. Three examples are the Hubble Legacy Archive viewer, the DESI Legacy Survey Viewer, and the SDSS Skyviewer.
The Portal team is working on an improved UI/UX for the Portal Aspect. We expect this to be deployed around the start of the LSST. It would be helpful if the UC could give specific examples of “improved interactive image viewer”, distinguishing between “better looking” and “additional features”. For example, are the desired improvements about “search” capabilities/UX or “results” (e.g., once you’ve found images, what can you do with them).
(3) Slack. LSSTC Slack is useful. The continued existence of LSST Corporation Slack after the transition to operations could be broadly helpful to the Rubin User Community. We suggest identifying a funding model to support this. This could be Rubin, NOIRLab, or LSST Corporation.
At this time, Rubin has concluded it can not support Slack as a global communication channel in Operations. Our main channel with the community is and will be Community.lsst.org.
(4) RSP development. The committee is concerned that some of the features of RSP@IDF will effectively be frozen-in with DP0.2/DP0.3. But substantial development in code, analysis tools, and the RSP-related environment will happen during commissioning. There will not be an updated RSP environment and associated data interaction until DP1, which is very close to ORR. There is the possibility of a 18 month gap between useful feedback on DP0.2/0.3 and DP1.
The RSP instance running at the IDF will not be frozen and will continue to be under continuous development. It will continue to receive new features as and when they are developed, tested and deployed. Feedback from users and from the Users Committee will continue to be taken. A presentation on planned development from RSP Team members will be given in the UC meeting at the Rubin 2023 PCW on Fri Aug 11.
D. Suggested topics for future UC meetings.
(1) RSP Development Plans. A report from the RSP team about future plans over the coming 12 months.
This is being arranged for the August UC meeting at the Rubin 2023 PCW. Rubin staff who are developing the RSP will prepare presentations and be present to answer questions.
(2) Reports from other centers. A report from other centers about user support to understand what has worked and not worked for different data centers trying to support different levels of complexity. Counterpart centers include IPAC, MAST, NOIRLab DataLab, and the NASA centers.
The CST is in touch with individual people at IPAC, MAST, and NOIRLab DataLab, and requests their feedback and/or reviews their materials at times to inform our own development. Requesting reports from these other centers is an approach we are not considering, but perhaps organizing a future symposium or workshop on user support would be possible. Attendees could be composed of developers and users of these and other platforms. This has been added to the draft agenda for the UC’s 2023B meeting.
(3) External data sets. Discussions about how Rubin users might integrate LSST data with other large data sets.
This has been added to the draft agenda for the UC’s 2023B meeting.
(4) Data rights. Further discussion of who will have access to Rubin data.
This is outlined in RDO-013. This topic has been added to the draft agenda for the UC’s 2023B meeting, please bring your specific questions about data rights to that meeting.