Title: General thoughts on formulating responses to the LOR call
Contributors: Alex Malz, Sam Schmidt
Co-signers: Jeff Newman, Shahab Joudaki, Maciej Bilicki, Johann Cohen-Tanugi
0. Summary Statement
The prompts in the LOR call aim to identify methods and metrics to ensure Rubin photo-z data products are useful to a wide array of extragalactic science analyses. These questions are not so dissimilar to those considered by the Dark Energy Science Collaboration (DESC) Photo-z (PZ) Working Group (WG) in the process of building its own photo-z pipeline for the specialized needs of a cosmological analysis. To jump-start discussion of setting science case-specific requirements within the broader Rubin community, we share in this informal post some general thoughts based on the PZ WG’s overall approach to addressing the same questions posed in the LOR call. This post does not constitute a LOR, nor does it preclude DESC members from submitting LORs to this call.
1. Scientific Utility
The science goals of each application guide the corresponding requirements on photo-z quality. Even considering cosmology alone, there are a variety of probes with differing sensitivity to photo-z quality as measured by diverse metrics, including those beyond the traditional bias, scatter, and outlier rate. Photo-zs from any estimator are in turn sensitive to prior information, be it non-representative training sets, incomplete template libraries, or errors in spectroscopic reference sets, as well as the quality of the data itself, encompassing the properties of the photometry and any number of choices in the data processing pipeline that produced it.
To ensure that photo-z requirements appropriate to each cosmological probe are met, DESC is developing software infrastructure, including the publicly available Redshift Assessment Infrastructure Layers (RAIL) code to stress-test photo-zs derived under a wide range of experimental conditions simulating systematics of priors and data. Though DESC will use it to characterize the response of cosmological constraints derived by the probe-specific analysis pipelines, the codebase is equally applicable to any science goal, and the development team welcomes non-members of DESC via the RAIL GitHub repository linked above.
2. Outputs
As Rubin photometry will encounter redshift degeneracy, i.e. galaxies with different SEDs at different redshifts can yield indistinguishable photometry, simple measures such as a photo-z point estimate and Gaussian error often fail to capture the redshift uncertainty landscape of cosmological samples. Thus, DESC’s primary photo-z data product will be redshift probability density functions (PDFs) that can propagate a more comprehensive notion of per-galaxy redshift uncertainty through a Bayesian analysis.
It’s worth noting that DESC’s plans are developed under the constraint that existing photo-z PDF estimators yield posteriors, though we’d prefer likelihoods. Photo-z posteriors from different estimators vary even in controlled experiments without sources of systematic error (Schmidt, Malz & Soo et al 2020) due to the influence of an implicit prior inherent to each implementation. Because the implicit prior is known to impact inferred cosmological constraints (Malz & Hogg 2021), DESC will produce photo-z posteriors from multiple methods along with information encapsulating their implicit priors so that likelihoods can be recovered in a Bayesian inference.
DESC will also make use of flags, e.g. for deblending, star-galaxy separation, and non-detections; these may not seem like a photo-z data product but should be considered at this stage, particularly because they may come “for free” as part of the photo-z estimation process. Considering the data processing steps at which relevant flags are obtained and used may help identify which to request as an essential output of Rubin Data Management (DM) in this call, especially for flags generated before photo-z estimation or used in an early step of science analysis, such as sample selection.
3. Performance
Requirements on photo-z accuracy necessary to achieve the required constraining power on the cosmological parameters by each cosmological probe are regularly updated in the DESC SRD, a living document that will continue to evolve along with the cosmology analysis pipelines upon which such probe-driven requirements are conditioned. In addition to the probe-specific requirements, the DESC PZ WG evaluates more fundamental metrics of photo-z posteriors intended to robustly assess photo-z data products in a way that is independent of the cosmological probe and analysis implementation; this can be advantageous when there are long-term plans to eventually implement a more advanced analysis approach that could have different sensitivities. Mathematical metrics of univariate PDFs beyond the “standard” bias, scatter, and outlier rates of point estimates are often better proxies for the response of a given analysis approach’s precision or accuracy to the choice of photo-z estimator and properties of the data or priors. It is valuable to investigate whether a given science use case’s requirements might be best defined in terms of such alternative metrics, such as those reviewed in Schmidt, Malz & Soo et al 2020.
4. Technical Aspects
Like Rubin overall, DESC is concerned with computational resources, particularly the footprint and accessibility of inputs and intermediate quantities alike, as well as the time and power necessary for training or sampling steps employed by some competitive photo-z estimators. DESC has few restrictions on coding language and has secured some human resources to help with scaling photo-z algorithms to billions of galaxies, but these nontrivial challenges may rule out certain estimators. It is nonetheless valuable to request that estimates from multiple approaches be provided, in spite of the limited storage resources, and the request isn’t impossible if one optimizes the storage parameterizations and number of parameters for nontrivial PDFs (Malz & Marshall et al 2018) to ensure the reconstructions from compressed table values meet science requirements.
DESC has critical if not universally quantitative requirements not only on photo-z end products but also on the inputs to any photo-z estimator, including both the quality of the prior information, e.g. completeness of template libraries or rate of erroneous training set redshifts, and the characterization of the input photometry, e.g. extended source photometry choices, rates of undetected blends and non-detections, or effectiveness of Milky Way dust extinction correction. Even if one can’t specify these requirements numerically in time for this call, it is well worth highlighting the systematics likely to affect a given science case.
5. What comes next?
This post is intended to spark discussion among consumers and producers of photo-z data products about how to approach this LOR call – it does not provide answers to the questions defining an LOR, neither to Rubin DM nor to those intending to submit LORs. We hope that members of previous, ongoing, and other future photometric surveys will also share how they approached these questions to provide the Rubin community with exposure to a variety of perspectives, ultimately contributing to the utility of Rubin’s photo-z data products.