Following the merge of DM-13388, the visit-level sky subtraction introduced in DM-9147 is now required by default for warping HSC data. This means that if you are processing HSC with an LSST-DM stack version after w_2018_05, you will need to run skyCorrection.py before running either makeCoaddTempExp.py or coaddDriver.py (which will require the use of sky frames) or disable the use of the sky correction in makeCoaddTempExp.py or coaddDriver.py. You can construct your own sky frames or download some I’ve prepared, which will soon be installed on lsst-dev (note that the full directory which contains calibs for the entire life of HSC is over 800 GB; for a smaller download, you could grab only the frames for the dates relevant for your data, but be sure to grab the calibRegistry.sqlite3 file and keep the same directory layout).
Building a sky frame
The sky frame is constructed in the same manner as biases, darks and flats. I suggest they should be created from wide-field survey observations (not from pointed observations with small dithers since bright objects may print through).
constructSky.py /path/to/data_repo --calib /path/to/calibs --rerun myRerun --batch-type=slurm --cores=140 --mpiexec="-bind-to socket" --time=0.5 --job sky-r --no-versions --id filter=HSC-R dateObs=2016-06-11 field=SSP_WIDE expTime=150.0
Note that this is a parallel process using ctrl_pool; you can either run it on a cluster (--batch-type=slurm or --batch-type=pbs, as appropriate) or on the local machine (leave off --batch-type, or use --batch-type=smp; be sure to adjust the --cores appropriately).
Having created a sky frame, it needs to be ingested into the calibs repo:
ingestCalibs.py /path/to/data_repo --calib /path/to/calibs --validity 180 '/path/to/data_repo/rerun/myRerun/SKY/*/*/*.fits'
Visit-level sky subtraction
This is performed after processCcd.py or singleFrameDriver.py, but before makeCoaddTempExp.py or coaddDriver.py. If you’re running meas_mosaic, this can run concurrently, as they don’t step on each other’s toes.
skyCorrection.py /path/to/data_repo --calib /path/to/calibs --rerun myRerun --job skyCorr-r --time 60 --cores 112 --batch-type slurm --mpiexec='-bind-to socket' --id visit=12345
Note that this is a parallel process using ctrl_pool; you can either run it on a cluster (--batch-type=slurm or --batch-type=pbs, as appropriate) or on the local machine (leave off --batch-type, or use --batch-type=smp; be sure to adjust the --cores appropriately).
Disabling use of visit-level sky subtraction
If you don’t want to use the visit-level sky subtraction (e.g., if you don’t have sky frames available), you can disable the use of the correction in the warps:
makeCoaddTempExp.py ... -c doApplySkyCorr=False
or
coaddDriver.py ... -c makeCoaddTempExp.doApplySkyCorr=False