Frequently Asked Questions
In this post, we provide responses for several frequently asked questions related to the Commissioning Announcement of Opportunity (AO). Additional questions are welcome. Questions can be posted either on this topic or via the following anonymous form. We plan to address as many questions as possible via this FAQ post and at the dedicated parallel session on this topic at the 2021 Project and Community Workshop, currently scheduled for Monday 9 August.
Q1: Why is the Announcement of Opportunity restricted to the US and Chilean communities?
A1: There was a separate call made for in-kind contributions from international contributors as part of the support for operations. That call included the opportunity to contribute to the commissioning phase as well. This Announcement of Opportunity is to invite “non-international” contributors, i.e., from the US and Chile, that were not part of the international call for contributions.
Q2: My institution/research group already have been working directly with the Rubin Project? Does my institution / research group need to respond to the Announcement of Opportunity?
A2: Section 3.2 of the Announcement of Opportunity describes Ex Officio members of the commissioning team, including several categories of non-Rubin-staff members of the Science Community who will automatically become members of the Commissioning Team in order for them to more efficiently carry out responsibilities associated with their designated roles. Individuals in these categories do NOT need to respond to this AO. In particular,
*“Groups who have already been performing work directed by the Project under collaborative or contracted efforts and who have made material contributions to construction effort will continue to be members of the Commissioning Team.” *
Please contact Chuck Claver and Keith Bechtol if you have questions about your specific situation. The Project plans to reach out to some individuals/groups with current working relationships to the Project to help clarify whether a formal response to the Announcement of Opportunity is needed.
Q3: What is the definition of a "group"? What is the relationship between a "group" and a Science Collaboration?
A3: There is not intended to be a one-to-one mapping between Science Collaborations and groups responding to the Announcement of Opportunity. The groups are not intended to “represent” a particular Science Collaboration. A group may include individuals involved across multiple science collaborations. Multiple groups consisting of members from a single Science Collaboration can respond to the AO. The guidance encouraging individuals with similar interests to respond as groups is partly meant to keep the number of points of contacts with the Project more tractable. See Section 5.1 of the AO for more info.
Q4: What does it mean for a "group" to formally sign an MOU with the Project?
A4: The MOUs are agreements between the Project and the home institution(s) of the individuals in the group. The MOUs would be signed directly by the individual who is the point of contact. It is NOT expected that the MOU needs to be signed by a department administrator / department chair. We expect the MOUs to be brief documents (<1 page text) primarily based on submitted Letters of Interest. The MOUs will name the participants and provide a few-sentence description of the planned contributions.
Q5: Regarding 'The MOUs will be assessed and re-evaluated on a yearly basis', what does it mean to re-evaluate a signed MOU?
A5: The yearly re-evaluation is primarily an opportunity to update the descriptions of value-added contributions, evaluate ongoing efforts, and review and update the list of personnel involved in the effort (in the event that specific individuals become inactive and/or new individuals are joining the group). The value-added contributions are not meant to be static and are expected to evolve throughout the commissioning period.
*The Project can decide to terminate a group’s or person’s role on the team status at any time due to poor performance or undue overhead. *
The participants may decide to end their own involvement with the Commissioning Team at any time.
Q6: The AO states that the 'anticipated total value-added contribution' is 15-20 FTE. Does that mean 15-20 FTE is the estimated needed level of integrated value-added effort? Or that it is what you expect to get?
A6: The Project will not rely on the contributions from non-Rubin-staff team members to fulfill core construction requirements and operational readiness criteria.
The 15-20 FTE integrated effort is a rough estimate intended to indicate the approximate scale the Project expects from this program that could be reasonably managed. The scale of the program is bounded by estimates for the level of interest and availability within the Science Community as well as by limiting demands on the Project to manage this effort (e.g., training, developing work assignments, oversight, and communication).
Q7: The 15 Example kinds of contributions seem to be more than examples but rather the specific areas where value-added contributions are being requested. Is that right? Does the Construction Project have any value-added contribution desires that are not Examples?
A7: The intent of these examples was to provide enough detail to help groups prepare their responses based on the skill set the groups have. The Project sought to provide enough examples to give a sense for the range of possible contributions. This list of examples is not exhaustive or exclusive. We encourage groups put forward their ideas for where their team could make contributions that would enable early science when operations start. The Project will help refine these concepts for maximum value within the commissioning effort.
Q8: Suppose a researcher does not currently have the resources to commit enough time as required by the AO and therefore has to wait for the Data Previews to provide feedback. Will there be enough time at that point for the Rubin Project Commissioning team to consider that feedback (and maybe collaboratively develop improvements) or is the Data Preview period mostly meant for the Science Collaborations to do "science" with the data?
A8: Given the schedule of the Data Previews with 3-month delays of data coming from ComCam and LSSTCam, there is tight time constraint for responding to feedback that would affect the commissioning effort. This said, the feedback from the community analysis of the Data Previews can have a significant impact on the algorithm development that will impact the first data releases - DR1 and DR2.
Q9: “Each non-Rubin-staff member/group of the Commissioning Team will be assigned a functional point of contact within the Project to help integrate their efforts.” Can you say a little more?
A9: The goal here is to facilitate collaboration between the Project and the group(s) providing the value-added contributions. The groups will have work assignments based on their proposed contributions. The functional point of contact on the Project helps to guide those efforts on a more day-to-day basis so that the effort is well aligned with the needs of the Project at a given time and the groups have the support needed to make their contributions.
Q10: How does high-level oversight of science performance work on the commissioning team? With early data, it is often not possible to divide up tests of data quality / features in the data into neat buckets. Is there an internal structure that connects areas of the commission effort? How should we think about responding to the AO in the context?
A10: The Project encourages individuals with a global commissioning perspective and experience who have the ability to triage/diagnose/disentangle complex features in the data to respond to the AO. When drafting the AO, we were thinking about how we would invite these types of contributions. A challenging aspect is how to phrase this type of contribution and how to evaluate proposals of this type fairly. In the end, we decided to opt for an approach of requesting more specific contributions, but making clear in the AO that once someone is part of the commissioning team, they would be welcome to participate in other areas of the commissioning effort.
We expect that individuals with this more global view / experience would also have specific focus areas that they could mention when responding to the AO. There is also a section of the AO document that describes “core competencies” (Section 5.7) that show how respondents can highlight their breadth of skills to address emergent issues as they arise.
Q11: For individuals who are already on the Project Commissioning Team (e.g., Rubin Observatory staff) that also have personal interests and collaborations with non-Rubin-staff related to the Announcement of Opportunity, should they be listed as collaborators on LOIs?
A11: Individuals who are already on the Project Commissioning team do not need to respond to the AO. If Rubin staff are interested in collaborating with individuals who are responding to the AO on particular value-added contributions, it would be advantageous to list them as collaborators on the Indications of Interest and Letters of Interest for context.
Q12: Could you point us to additional information/documentation regarding the Data Previews that would help to understand the difference between being part of the Commissioning Team via the Announcement of Opportunity versus working with the Data Previews only? For example, opportunities for iteration and feedback with the Project? Access to lower level data? Timescale for implementing improvements?
A12: The primary difference between Data Previews and Commissioning data access is twofold: First, is the timeliness of access to the data, where the commissioning team will have immediate access and the Data Previews will be subjected to a 3-month delay; and Second, the goals of the two efforts are different - the commissioning data analysis is aimed at demonstrating operations readiness and construction completeness and the Data previews are aimed at preparing the science communities for understanding and analyzing the LSST science data products.
See also Q17, Q18, and Q19.
Q13: How long in advance before a given Data Preview would participants in the AO have access to the data?
A13: Community members who are selected as part of the Project Commissioning team will have immediate access to the data in real time. Data Previews are put out by Operations Team with a 3-month delay at the conclusion of observing with ComCam for DP1 and LSSTCam + Science Validation surveys for DP2. The Operations Team will need to make sure data are ready to be released and resources are available for community investigation and interaction. There is no connection between the response to this AO and the access to Data Previews.
Q14: What raw/calibrations/telemetry data products will be made available as part of the Data Previews to support more technical investigations (of algorithms, for example) by members of the science community? More concretely, do you anticipate that raw image data will be available, including associated metadata, and calibrations? Same question for data from the guiders and wavefront sensors on the main camera, and also AuxTel data? Is there a document where we can read up in more detail about the data included in the Data Previews?
A14: At a minimum, the calibration data products used to make the DP data products will be made available through the RSP. Raw images and meta data corresponding to the data products will also be available.
Guider and WFS data and other system telemetry will be available. WFS images will be available from the RSP and telemetry from the Engineering Facilities Database (a copy of this data base will be available at the USDF for user access)
Some AuxTel data products will be available.
We are working on a descriptive document for the data releases including DPs and will make it available as soon as we can.
Q15: What is the timescale for the Project providing further guidance/policies on communication regarding commissioning data beyond the Project Commissioning Team?
A15: This will require a deeper conversation with the System Integration Test and Commissioning (SIT-Com) Leadership Team in drafting a concrete policy for how commissioning status and results are discussed - both internally and externally.
Q16: Can you say more about the role of the selected photo-z teams that are Ex Officio members of the Commissioning Team?
A16: The intent is that individuals who are directly involved in the implementation of photo-z estimators would become Ex Officio members of the Commissioning Team so that they can begin the implementation and testing with on-sky data from ComCam and LSSTCam as early as possible. The focus of this effort is applying the algorithms to on-sky data and generating photo-z estimates that are ready for more comprehensive science validation and comparison that would occur with the Data Previews. Project Data Management will take a lead role in defining these photo-z teams based on the needs to implement photo-z algorithms for LSST.
Q17: Will there be an opportunity for feedback and response during the Data Preview era? What are the mechanisms for members of the science community to provide feedback to Rubin DM on commissioning data if they are only working with the Data Previews (i.e., they do not directly participate in the Commissioning Team)?
A17: We will be developing and using the community support “system” for the full survey in pre operations.
Fundamentally this support model is based on community.lsst.org. We expect broad feedback through that channel which the Community Engagement Team will be monitoring and in some cases moderating/driving. Issues that require more in-depth discussion and technical answers will be taken by the CET team to experts within Rubin. Answers and solutions will be propagated back out to the community through community.lsst.org.
The formal venue for user feedback about the Rubin data products and services will be the Users Committee (UC). The UC is charged with soliciting this feedback from the community and reporting to the Lead Community Scientist and the Rubin Observatory Director with recommendations. The UC charge is being finalized and the committee should be in place by September 2021. The UC’s work will be facilitated by the CET, and they will focus on the DPs at first. The UC will stand throughout Rubin Ops.
Science Collaborations will continue to have construction project liaisons that will participate in SC meetings. This effort will be evolved into a formal SC Liaison program facilitated by the CET throughout Operations.
Some Science Collaborations will likely have very specific proposals or input for how the pipelines might evolve to ensure science requirements are met. We have only an outline for the process formally for how to do this during survey operations, but we recognize the need and will be working on this. Indeed, we can imagine designing, implementing and commissioning our process for evolving the science pipelines via a carefully controlled pathfinder experiment prior to DR2, using input from analysis of the data previews and DR1. The DR1 dataset will be generated using the construction project-built science pipelines. During pre-operations, the construction project will continue to accept input from the science collaborations on an informal basis, and also to make use of information provided by the operations team.
For how the latter will work with DESC in particular, we wrote in our DP0 MOU that “Rubin will set up the DRP-like re-processing pipeline to generate the DP0.2 products … [to] be close to the delivered pipeline from the Rubin construction project, but its configuration may incorporate input from DESC on a best-effort basis. The Rubin [operations] Data Production team will collaborate with the DESC members who were involved in the image processing effort to formulate this input in advance of DP0.2 re-processing start, and then again on the set-up and execution of the workflow (from which experience DESC hopes to improve its own processing).” Beyond DP0.2, we will continue to incorporate input from the science community on a best effort basis, with collaborations developing out of interactions started on community.lsst.org.
Q18: Will there be adequate time and opportunities to provide feedback on the LSST Science Pipeline data processing algorithms as part of the Data Previews, so that the feedback could be incorporated into the first DR? If the time is very short, there would be less opportunity for feedback and it would be more important to encourage and enable commissioning contributions.
A18: At this time, we do not plan to significantly modify (i.e. through significant user feedback) the pipelines between the DP2 and DR1. Given the uncertainty in our schedule to complete commissioning/SV and start operations, it is likely the time between the SV phase and DR1 will remain uncertain and perhaps fairly short. We believe it will be most efficient to get through DR1 and then implement the process (still TBD, as above in Q17) for how to routinely evolve the science pipelines based on the best input we are getting for the community (or internally). We recognize that much of the community will be focused on DR2 as the first full area data release, so see the note above about starting to implement our pipeline evolution process such that some changes could be accepted prior to freezing the DR2 pipelines.
Q19: How do you envision the feedback loops working during the Data Previews? There were some ideas discussed by DESC members that would require a rather quick turn-around for tests and feedback. If the turn-around is anticipated to be rather slow, it might be important to understand some of these questions during commissioning. Another possible feedback loop could be if members of the science community have the opportunity to provide input to the Rubin team’s V&V effort during and even after commissioning.
A19: See the answer above for Q18. Key inputs the community can make, based on any of the data preview data available to it, are validation tests that the Rubin V&V teams can pick up and implement. The Commissioning V&V team will lead the way, but the Operations V&V team will be shadowing them, and extending, further automating, and documenting the tests that they will inherit, in preparation for survey operations. We expect this community-observatory interaction to then continue through the survey, again mediated by the Community Engagement Team and based on community.lsst.org and feedback via the Users Committee.
Further to this, DP1 will be based on ComCam data and the priority will be to continue system verification activities. We expect DESC members on the commissioning team to play important roles in this channel of feedback, and those looking at DP1 less so. The focus of the Rubin team will quickly shift to the full Camera commissioning.
SV will be complete at handover to Operations. SV data will be processed by the Operations team and won’t come out until after survey operations have started. DP2 is based on SV data. This goes back to the point above. We will be putting most of our effort into getting DP2 out and simultaneously beginning processing for DR1. It is unlikely significant changes to the pipelines, except those needed for basic requirements, will be made in this period. It will be about bug fixes and scaling to full DR processing.