Minimum necessary files to produce mean template

Support
#1

Hi support

I am processing imagery of a field with the final goal of producing a template from which to subtract from the individual warps. My steps to making the templates: processCcd > makeDiscreteSkyMap > makeCoaddTempExp > assembleCoadd

All works fine and dandy.

My question is what intermediary data products are not explicitly necessary for production of the templates?

For example, after processCcds and makeDiscreteSkymap, can I compress (or nuke) the SRC and ICSRC files or are they needed by make/assembleCoadd scripts?

Thanks for the help!

#2

Hi Wesley, You need to keep

  • the skyMap (deepCoadd_skyMap)
  • Direct Warps (deepCoadd_directWarp)
  • and also need the PSF-Matched Warps (deepCoadd_psfMatchedWarp) if you plan on doing any image subtraction or if you’re using CompareWarpAssemble (which as the name implies subtracts warps from template to find outlier).

But depending on how you plan to run the AssembleCoaddTask or CompareWarpAssembleCoaddTask you may still need to keep the calexps around too for their WCS’s. When you run the commandline task, the default image selector loops through the calexps you supply as arguments to --selectId, read the WCS and bounding box to determine if they overlap the patch you’re interested in.

You can still run the Task and generate a coadd from the warps, however, by calling it yourself. I do this all the time when I need to generate a coadd from some warps and don’t have the calexps available:

import lsst.daf.persistence as dafPersist
import lsst.pipe.tasks.assembleCoadd

# data I downloaded:
butler = dafPersist.Butler('/project/yusra/reproduce/hscdata.mtk.nao.ac.jp/hsc_ssp/dr2/s18a/data/s18a_wide') 

visits = [130366, 130364, 130354, 130334, 130074, 130044, 130042, 130012, 129992, 128594, 128572, 119406]

dataId = {'tract': 9469, 'patch': '1,6', 'filter': 'HSC-I'}

warpRefList = [butler.dataRef('deepCoadd_psfMatchedWarp', visit=v, **dataId) for v in visits]

dataRef = butler.dataRef('deepCoadd', **dataId)

config = lsst.pipe.tasks.assembleCoadd.CompareWarpAssembleCoaddConfig()
config.doSigmaClip = False
config.subregionSize = (10000, 200) 
config.removeMaskPlanes.append("CROSSTALK")
config.doNImage = True
config.badMaskPlanes += ["SUSPECT"]
config.assembleStaticSkyModel.subregionSize = (10000, 200)
config.doAttachTransmissionCurve = True
config.interpImage.transpose = True # Saturation trails are usually oriented east-west, so along rows
config.coaddPsf.warpingKernelName = 'lanczos5'

task = lsst.pipe.tasks.assembleCoadd.CompareWarpAssembleCoaddTask(config=config)

result = task.runDataRef(dataRef, warpRefList=warpRefList)

If you’re happy running it with your own script ^^^, you can nuke all the other intermediate data products except the warps and skymap.

PS: plug for our upcoming Gen3 middleware system that we’re in the process of converting to… Tasks are now required to explicitly declare their inputs and outputs. You can find this info in their “Connections” class.

#3

Fantastic Yursa! Thanks!