Proposed date: Tuesday August 1 12:30 pm to 2:00 pm Pacific.
Connection: https://bluejeans.com/426716450
Suggested audience: DM Science Pipelines
Requirements for Coaddition Tasks Redesign
I’m gathering a list of stakeholders and requirements for a coaddition redesign. A number of use cases have popped up recently that demonstrate that the current hardcoded assumption that we’ll run assembleCoadd once with one config is insufficient, and the time to update it is now. You may have noticed that the number of coadd data dataProducts for every configuration of coadd we plan to generate (e.g. deepCoadd, goodSeeingCoadd, deepCoaddPsfMatched, deepCoaddLikelihood etc…) have been proliferating in our obs packages. The main feature of this redesign is to move these coadd descriptors (which signify configs with which assembleCoadd was run) from the data product name into the data ID.
Stakeholders and Use Cases
The following tasks are awkward given the current framework and could benefit from a redesign. I’d like to make sure the the design meets the needs of these use cases:
-
Make direct/psfMatched, deep/bestSeeing etc… coadds in one DRP (without --clobber-config!). In LDM-151 these are called
DeepCoadd,BestSeeingCoaddandConstantPsfCoadd. We may want different modelPsfs for different purposes (e.g. deblending, artifact rejection) too. -
Implementing artifact rejection in deepCoadds. This requires a PSF-Matched Median coadd as an intermediate data product.
-
Generating the series of
ShortPeriodCoadds. @jbosch indicated that @boutigny was working toward creating these now. -
TemplateCoaddsfor Alert Production binned by parallactic angle or wavelength to account for DCR. @isullivan -
Your use case here.
##Sketch of Coaddition Interface Changes:
Because it is easier to find gaps in a prototype design than recalling your requirements to me, here’s the strawman schematic of what I’m envisioning:
1. Designate coadd types in data IDs:
Proposed Keys (analogous to “patch” or “filter”):
"warpType": ('direct', 'psfMatched', 'likelihood')
"seeingCutoff": ('deep', 'goodSeeing')
# For ShortPeriodCoadds:
"baseline": ("y1", "y2", "y3", ... , "y10")
# For TemplateCoadds:
"wavelength": (?)
"PA": (?)
On obs_base the coadd data products could collapse into one for every parent task (see below):
coadd:
persistable: ExposureF
storage: FitsStorage
python: lsst.afw.image.ExposureF
template: coadd/%(filter)s/%(tract)d/%(patch)s/%(seeingCutoff)sCoadd%(warpType)s.fits
level: Skytile
to be retrieved like:
butler.get("coadd", filter='g', tract=0, patch="0,0", warpType="direct", seeingCutoff="deep")
2. Add parent coaddition task
There is still a problem of how to generate the full suite of coadds without using --clobber-config.
Currently, a single DRP only allows you to run assembleCoadd.py once with with one config. To make multiple coadd types with different configs, they must be output in different repos. In general, Pipetasks take an input data ID and a config to produce an output data product (with dataIDs constructed from the input --id). The input and output data product is implicit in the Task. The input (and sometimes output e.g. makeCoaddTempExp) data ID is specified by the user.
Proposal: Add a parent coaddition task “CoadditionTask” (to be replaced by a supertask I’m sure) that takes the dataIds you want to produce. It’ll make the appropriate calls to the subtask assembleCoadd. The call signature would look like:
coadditionTask.py input/repo \
--id filter=g tract=0 patch=0,0 warpType=psfMatched^direct seeingCutoff=deep
To make this backwards compatible, warpType=direct and seeingCutoff=deep can become implied defaults so that users in the habit of leaving those off still can.
This will not preclude the existence of other parent coaddition tasks, such as a “makeTemplateCoadd.py”
Questions for Discussion
I’ve left out a number of details to be discussed. For example, where and how are these Coadd Keys defined? How does a developer add a new Coadd Key? How flexible? How does CoadditionTask convert these keys into configurations and calls to assembleCoadd?
TL;DR The coaddition task interface is undergoing a redesign. Please comment if you’d like to attend the RFD or have requirements, use cases or ideas you’d like to share.