Meeting for RFC-152 was held 12pm Pacific 2016-03-02. Attendees: @jbosch, @RHL, @ivezic, @ktl, @timj, @hsinfang, @jdswinbank
The follow is @jbosch’s original agenda, annotated with @timj’s notes, after further adjustments by @jbosch.
AGENDA
-
Preview Big Questions
-
Go through abstract Preprocessing Stages quickly (not the focus of this meeting)
-
Go through abstract Pipeline Stages, thinking about Big Questions
-
Revisit Big Questions
-
Go through the proposal for concrete pipeline stages on DM-4839
BIG QUESTIONS
-
What have I missed (or incorrectly assumed)?
-
What do we need to build the PSF star catalog, and should we reuse the one from the last DR?
-
Do we need to do any chromatic-aware processing in Image Characterization? If so, how/when do we revisit it once we have colors?
-
How do we build artifact masks for coaddition, and do we do that before or after we build templates for image subtraction?
-
What kinds of coadds do we build?
-
Is there any way we can remove the big MOPS sequence point?
PREPROCESSING STAGES
(do these before processing the main wide survey)
Calibration Products Pipeline
- Generate typical ISR inputs.
- Generate brighter-fatter kernels.
- Generate wavelength-dependent photometric corrections
PSF/Wavefront Characterization
- Use dedicated observations taken at various focus levels to constrain relationship between wavefront sensor measurements, telescope telemetry information, and star images. I think of this as computing some sort of basis function set for PSF modeling in wavefront space, but I’m waving my hands.
Sensor Distortion Characterization
- Use a set of many dithered observations (probably on a moderately dense field) together with flat fields to constrain frozen-in-sensor coordinate system effects (tree rings, pixel area variations). Could use the full survey dataset for this, but probably don’t need to. Could just be data from deep-drilling fields.
Deep/External Processing for Bayesian Priors
- Process deep-drilling fields (and possibly joint-process with other surveys in overlap regions) first, so we can use measurements (e.g. distribution of galaxy ellipticities) as a prior on the wide survey.
- Processing otherwise uses the same procedure as the main survey, but we need to figure out how to bootstrap priors (probably just using less informative priors in deep fields).
PIPELINE STAGES
Image Characterization
- Includes ISR, snap-combine, first-pass backgrounds, single-frame astrometric and photometric calibration.
- Includes some PSF estimation, but possibly (probably?) not the final one.
- Does not necessarily include detect/deblending/measurement for final Source table, if that requires better PSF, background, or calibration.
- No chromaticity here; if we need e.g. chromatic PSFs at this stage, we need to find a way to get colors at this stage.
- @rhl is concerned this might end up being necessary.
- Maximum sequence point scale is a single visit, but most work only needs a single CCD. Probably scatter-gather between visits and CCDs (but maybe multiple cores working on a single CCD, too).
- @rhl: don’t assume that we can approximate this as CCD-by-CCD processing; full-visit stuff is very important.
Joint Calibration
- Fit for best astrometric and photometric calibration for all exposures overlapping a region of sky (area of a handful of focal planes?)
- Lets us constrain spatial variations on smaller scale than reference catalog allows. Expect Gaia to constrain larger scales.
- First opportunity to estimate colors for objects; if we don’t use a PSF star catalog from a previous DR, this is our first opportunity to construct one.
- @rhl is concerned that the PSF star catalog we construct at this stage might not be good enough; would be better to build one using classifications from a better PSF.
- Sequence point for all visits overlapping a region of sky. Most work is probably multithreaded linear algebra, but I/O could also be a bottleneck if we don’t parallelize it intelligently.
Final PSF Estimation
- Needs PSF star catalog for better S/G separation, colors to use for chromatic PSF determination.
- Maximum sequence point is probably a single visit (some thoughts about using galaxy difference images, but that’s probably not a baseline kind of idea). Most work is probably on a visit scale, need to think about how to parallelize effectively within that.
Coaddition
- Lots of kinds of coaddition, may not want to do all of them.
- Direct coadds with discontinuous lazy PSF - lossy, but less lossy than PSF-matched. Can’t do outlier rejection. Can update PSF model without updating the coadd itself.
- PSF-matched coadds - most lossy, but with simple, continuous PSF. Needed for aperture and moments measurements to be consistent across the sky and across bands. Can do outlier rejection.
- Matched-filter coadds - not lossy, but only directly usable for detection and point source measurement. Can’t do outlier rejection.
- Decorrelated coadds - optimal, but unproven with lots of algorithm development needed, with continuous but complex PSF; need some new tradeoff to deal with computational complexity or (in Kaiser coadd limit) unrealistic assumptions. Can’t do outlier rejection.
- Decorrelated and PSF-matched - much less lossy PSF-matched coadd, derived from decorrelated coadds. Unproven, lots of algorithm development needed. Can’t do outlier rejection.
- All coadds but PSF-matched require artifact masks derived from some sort of difference imaging (maybe background matching).
- Probably want coadds across bands for detection matched to more SEDs (chi^2 is just a special case for sky SED?). Need to decide whether to actually use different PSFs for different hypothesis SEDs; will need to do something like this for diffim templates, at least.
- Also need coadds for different epoch ranges, to detect faint slowly varying or moving objects.
- Sequence point for all CCD images (possibly visit images) that overlap a patch of sky (smaller than region needed for Joint Calibration). Most processing on visit scale before gathering epochs.
Background Matching
- Estimate (N-1) backgrounds on N exposures that overlap a patch of sky.
- Probably should happen during coaddition (many shared operations).
- Could be used to find some artifacts, since we produce difference images (but after warping all science images to coadd coordinate system, and with weaker requirements on matching kernel quality). Not clear if we can use this to find all artifacts (ideally want to find them before warping).
- Probably a sequence point for all visit images that overlap a patch of sky (and this is why coadd production probably needs full visits rather than CCDs).
Source Measurement
- If this needs final PSF estimation, calibrations, and/or backgrounds, it needs to happen after those are produced.
- Maximum sequence point is a single CCD image.
- @jbosch: Not sure what science requirements are.
- @rhl: Need to keep it until we demonstrate multifit can, for example, do astrometry
- @jbosch: Might be significant savings for database system if we can drop it
- @ktl: Definitely.
- @rhl: How do we get colors to this stage to do chromatic processing? Problematic.
Image Subtraction
- Needs template coadd as an input.
- Probably needed to generate artifact masks for coaddition.
- @jbosch: Could find CRs in snaps, but other sharp artifacts could be hard to find in background-matching, which is post warping.
- @rhl: Don’t always assume we will have snaps
- @ktl: Would we get rid of snaps without having alternate CR rejection system?
- @rhl: We have template images in DRP.
- @jbosch: Might be harder for alert processing.
- @rhl: Read noise issues in u-band may be an issue; could drop snaps to improve SNR
- Unlike Nightly, could use PSF models as prior for kernels or do full-visit kernel modeling some other way.
- Maximum sequence point is probably individual visits; maybe CCDs if we just use per-CCD matching kernels.
Coadd Detection
- Find sources on a suite of coadds - generates Footprints with Peaks, not deblended sources.
- Probably runs on Matched-Filter coadds.
- Sequence point is just a suite of matched-filter coadds for different SEDs and epoch ranges in a patch.
- @ktl: Please clarify which of these steps depend on explicit ordering.
- @jbosch: That’s in the concrete proposal; this is just a weak ordering. But coadd detection does not depend on image subtraction any more than coadds themselves do.
MOPS
- Associate fast-moving object detections from Image Subtraction with existing solar system objects.
- FULL SURVEY SEQUENCE POINT (!?)
- Unless we really want to start each DR from scratch, will want to update an existing solar system database - possibly tricky since all of the images in the DR will have already contributed to the nightly MOPS database using different processing.
- @jbosch: Using the existing database as a starting for an iterative procedure is probably fine, if we don’t use it to inform final result. Otherwise statistical double-counting.
- @ktl: Big worry is asteroids that disappear in DRP but were in L1 nightly
- @timj: What is threshold for acceptance at MPC?
- @rhl: Probably being too squeamish. Guess is this is not a real problem
Association and Deblending
- Associate and merge peaks from coadd detection, image subtraction, and MOPs to form Object Candidates.
- Reject redundant Candidates and deblend into Objects using suite of coadds.
- Might be able to split this into a pure catalog association operation followed by image processing, but I’m not sure there’s value to that.
- Sequence point is over a patch of sky, but with a potentially large suite of coadds. Need to find a way to parallelize within that scale (pixels? object families?)
Coadd Measurement
- Run measurement algorithms on coadds, using deblended pixels. Probably need a suite of coadds (no larger than suite needed for deblending, possibly smaller).
- Sequence point could be a patch of sky with a suite of coadds. Could also parallelize independently over object families, but determining the extent of those families prior to running measurement is problematic.
- @jbosch: Need to fit across bands.
- @rhl: Worried about large blends. Can you define “isolated” ever if you are deep. Maybe larger than 4k x 4k object families. Galactic centre would be too large.
MultiFit
- Fit models to individual exposures using coadd measurements as starting point. Should be able to use coadd measurements to put hard bounds on region of each exposure needed for fitting a particular object.
- Ideally parallelize over object families, maybe over pixels within that.
- Probably need all bands in memory simultaneously.
- May need some sort of divide-and-conquer for large blends.
- Need clever algorithms for iterating over objects in a way that’s I/O and CPU-throughput friendly (especially given large dynamic range in object family size).
- @timj: Does multifit handle proper motion fitting?
- @jbosch: Astrometry community doesn’t necessarily believe we’ll pull it off, but we’re either braver or more foolish. MultiFit will fit at least PM and Galaxy shear.
- @ivezic: A simple demonstration would be very useful.
- @rhl: Need a real dataset but may not have enough of a sample yet. Maybe Phosim can generate reasonably “bad” data.
Forced Photometry
- Measure fluxes at the position of every detection in every exposure, holding everything but amplitudes fixed.
- No obvious reason this couldn’t be done during multifit, but it could also be done by parallelizing over CCDs and asking DRP DB for reference catlaog that overlaps it.
Missing Stuff
- @rhl: Bright star masks
- @ktl: Building epoch-based coadds is mentioned in DPDD. Do these have overlaps in time? In which case you can’t coadd the epoch ones into the full range.
Return to Big Questions
- Can we reuse PSF star catalog from previous DR?
- @rhl No question it is the wrong thing to do to reuse, but maybe it doesn’t really matter.
- @jbosch: I like the simplicity (in e.g. provenance) of not reusing it. I think we need multiple PSF estimation stages anyway, so no cost to building it from scratch for each DR.
- @rhl: Should start with at least Gaia (and WFIRST, etc) catalog, even if we don’t reuse our own DRs.
- @jbosch: Definitely.
CONCRETE PROPOSAL
Reviewing https://github.com/lsst-dm/drp-docs/blob/master/tasks-top.yaml
General Principles:
- Current proposal tries to minimize large-scale iterations if we think we can get away without them.
- Should switch to preference for adding any large-scale iterations we think might be necessary; can triage later.
Specific Recommendations:
- Iterate between Joint Calibration and Final PSF Estimation.
- Iterate between coaddition and image subtraction to generate better templates.
Closing Discussion
- @ktl: What more is left for the working group on DRP?
-
@jbosch: I’ve been using YAML, because I want to think of it as a flowchart without doing the formatting. Is this a good idea? Is EA better?
- @ktl: Some advantages, but could be hard to extract from EA once it’s in. Would help with relation to ops planning but probably not relevant. No compelling reason at this time.
- @timj: Graphviz? OmniGraffle Pro can read .dot files.
- @jbosch: Will continue with YAML, will hope someone writes a graphviz script for figures later.
-
@ktl: Where do other types of exposures fit in? Twilights, different exposure time? Deep-drilling?
- @rhl: Coadding different exposure times complicates things. Otherwise processing is similar.
- @ktl: Any new requirements for other subsystems?
- @ivezic: Multifit was main driver for sizing model. Are all these other iterations minor in comparison?
-
@jbosch: Do I next update design documents or drill down on detail?
- @ktl: Narrative text is easier to comment on.
- @jbosch: LDM-151 is a possibility but would make it a larger document.
- @ktl: Okay to rewrite it in whatever way is required is fine. Can be multiple documents.
- jbosch: Makes sense to have discrete documents for AP, DRP, CPP (calibration) and SDQA with LDM-151 as an overview paper.
- @ktl: Use latex or RST.
- @jbosch: Need to sort out bibliographies in RST first.