DIA processing between DRP[1] and AP[2] will be different. Calibrations of PVIs[3] will be better in DRP, and templates will potentially be updated. The difference will likely be greatest in the Year-One Alerts and the Data Release 1,2 processing of the first 6,12 months of the survey.
Even meeting the same purity and efficiency requirements may not result in the same set of objects detected between AP and DRP. How much of a difference should the users expect?
Will there be a matching between AP DIAObject[4] IDs and DRP DIAObject IDs? Will this matching be provided by Project+Operations or will it be up to interested science groups to do this themselves?
How will the DIAObject ID labels be different between AP and DRP. Will they be unique between AP and DRP. Will they be unique across data releases?
Does it make even sense to refer to the set of AP DIAObject IDs “within” a Data Release? AP doesn’t formally fit into a Data Release. AP will of course be run between data releases, and it doesn’t make sense to think of AP on an image as “belonging” to a data release prior to that data. But it doesn’t clearly make sense for AP to be considered part of the next Data Release, either.
Will DRP provide forced-photometry on all AP DIAObjects? Imagine, e.g., an object that was detected, studied, and somebody went and took a spectrum and confirmed it as an interesting FoozaWhatsIt. But imagine that in DRP, the masks are grown a bit to match where photometry and detection are more reliable and that object is no longer detected. Will this object be referred to only by its AP DIAObject ID?
Perhaps the former question is really asking: what’s the relationship between Rubin Observatory/LSST DIAObject IDs and community transient and variable clearinghouses, such as the IAU SN, IAU MPC[5], or the Transient Name Server tracking?
[The intent of this topic is to summarize and generalize a discussion that occurred over Slack.]
[1] DRP: Data Release Processing
[2] AP: Alert Processing
[3] PVI: Processed Visit Image. Also referred to as a calexp.
[4] DIAObject: Difference Image Analysis Object. The result of associating individual detections on subtracted image.
[5] MPC: Minor Planet Center.
I don’t think it’s possible to provide a 100.00000% accurate mapping, so we will provide unique names in each data release. Obviously you can do a 9X.Y% mapping that is very useful but it requires decisions about magnitude priors and such like that Rubin DM knows no more about than the community — and that’s a reason not to do it on project.
I don’t think it is possible for DRP to independently re-detect every DIASource/DIAObject found by AP. The processing is independent, so we shouldn’t expect 1-1 correspondence in a wide variety of circumstances (see DPDD section 3.3 for some discussion).
However, it is certainly possible to force photometer AP DIAObjects on all DRP data, which I expect will provide the information you are after. This isn’t planned for all DIAObjects by default, but if you are only concerned about objects with spectra (or a larger set of SN-like candidates) my sense is that this would be quite practical in user processing.
All DIAObject IDs will be unique across all AP alert, AP Prompt Products Database, and Data Release (DMS-REQ-0292 (or see Sec 2.4 of the DPDD)). For this reason any astrophysical object will end up with several Rubin identifiers by the end of the survey. I expect a TNS-style naming approach external to Rubin will be necessary and valuable for many science cases.
I expect brokers and/or science teams may report objects of interest to the Transient Name Server (https://wis-tns.weizmann.ac.il/), where they’d be given stable AT/SN names as appropriate and associated with detections from other surveys. I don’t think such reporting is an activity for Rubin DM.
I suspect that many analysis groups will find that they have to internally translate their objects of interest into a stable name; Changing object names every year may be too impractical for analysis.
[While individual science groups will need to keep their own lists, it does seem like there should be a scale that is less than “all of Rubin/LSST DIAObjects” which is sufficiently useful and less prone to decisions based on particular science goals.
All Objects detected in that DR, which includes all DIAObjects detected in that DR, possibly with some deduplication between DIAObjects and Objects detected on coadds, to the extent we believe we can do that robustly.
I believe it does not include SSObjects, which are defined to be disjoint from DIAObjects - there may be some similar follow-up processing for those that I’m not as familiar with off the top of my head.
Thanks so much to @mwv for summarizing these questions which arose in Slack but are certainly appropriate for this Data Q&A topic. A response which reflects the current state of DM’s plans has been prepared, and additional discussion and follow-up questions are welcome.
The Populations of AP/DRP DIA Objects: Users should expect some differences between the sets of real time-variable DIA Objects identified by the AP and DRP pipelines. There will be real transients and variables that are detected and included in the DIA Objects table of one pipeline but not the other. For example, faint sources at the detection limit might have their signal-to-noise ratio shifted slightly above or below the detection threshold in AP or DRP, depending on the template images, conservativeness of masks around bright and saturated stars real/bogus classifier, etc. Such instances will be quantified and characterized by verification and validation processes, but there is currently no simulation or estimate for Rubin data to which we can point users at this time.
DIA Object Identifiers: All DIA Source and Object IDs will be globally unique across all pipelines and data releases. The Project will not provide a direct matching of DIA Object IDs between AP and DRP (e.g., S.3.3 of ls.st/dpdd), nor between data releases (e.g., S.2.4 of ls.st/dpdd). The AP DIA Objects should be considered as independent of the DRP DIA Objects (and as a continuously-generated data product), and not as a part of any Data Release. It is left for the scientific community to decide whether and how Rubin identifiers are incorporated into community platforms and databases (and Target Observation Managers).
DRP Forced Photometry: It is not the plan for the DRP pipelines to perform forced photometry for all AP-detected DIA Objects, but it will do this for all DRP-detected DIA Objects. (More information about DRP forced photometry can be found in S.4.2.4 and S.4.3.3 of ls.st/dpdd.) The Rubin Science Platform will provide users the ability to generate DRP DIA forced photometry at any given sky location, which would accommodate the example of a spectroscopically-classified FoozaWhatsIt
