I’ve just merged DM-9147, which contains support for improved sky subtraction. This is an implementation of the sky subtraction algorithm advocated by the Hyper Suprime-Cam team at NAOJ. The algorithm incorporates two elements:
- An empirical background model extending over the entire focal plane. This model operates across CCD boundaries, meaning that discontinuities at CCD edges are avoided. The larger scale this can be (without leaving variable features), the better since this scale sets the scale at which astrophysical features are over-subtracted. Experiments indicate that the appropriate scale for HSC is 1024 pixels (a scale of 2k pixels leaves features that vary from exposure to exposure).
- A scaled “sky frame”, which is the mean response of the instrument to the sky for a particular filter. This allows subtraction of static features that have a smaller scale than the empirical background model.
Here’s an example of the new sky subtraction algorithm, a coadd (Cosmos field, r-band, tract=9813, patch=5,5) constructed using the sky correction (left) and the same processed with vanilla hscPipe 5.0-beta4 (kinda LSST 13.0; right). The dark (negative) halos around bright objects in the old processing (right) are not present, and the wings are much more visible with the sky correction turned on (left).
How to use
The sky frame is constructed in the same manner as biases, darks and flats. I suggest they should be created from wide-field survey observations (not from pointed observations with small dithers since bright objects may print through). Here’s an example command-line:
constructSky.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-sky --batch-type=slurm --cores=140 --mpiexec="-bind-to socket" --time=0.5 --job sky-r --no-versions --id filter=HSC-R dateObs=2016-06-11 field=SSP_WIDE expTime=150.0
Having created a sky frame, it needs to be ingested into the calibs repo:
ingestCalibs.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --validity 3650 --mode=link '/tigress/HSC/HSC/rerun/price/dm-9147-sky/SKY/*/*/*.fits'
Application of the sky subtraction algorithm is done using skyCorrection.py, which writes a background model that replaces the background previously removed as part of single frame processing and removes an exposure-wide background (1000 pixel-sized super-pixels) and a scaled sky frame. Note that this doesn’t modify anything, but only writes a new background model (dataset type skyCorr):
skyCorrection.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --job skyCorr-r --time 60 --cores 112 --batch-type slurm --mpiexec='-bind-to socket' --id field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2
In order to have the sky correction applied during warping, you need to set doApplySkyCorr=True, e.g.:
coaddDriver.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --id tract=9813 filter=HSC-R --selectId field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 --job coadd-r --time 100 --cores 16 --batch-type=slurm --mpiexec='-bind-to socket' -c makeCoaddTempExp.doApplySkyCorr=True
Bonus features
-
skyCorrection.pyalso writes a mosaic image of the entire focal plane with the sky correction applied; the dataset type iscalexp_camera. - The various calib construction tasks (e.g.,
constructBias.py,constructFlat.py) now also write a mosaic image of the entire focal plane; the dataset type is<calib>_camera.
Example sky frames
Here are some HSC sky frames in the (left to right, top to bottom) g, r, r2, i, i2, z bands. The rings in the r and i bands are due to radial variations in the filter curves. The g and r2 bands show some CCD-dependent features probably due to color terms, and there are some individual amplifiers visible for reasons we don’t yet understand.
