Hi,
I would like to stack HSC images which were processed by hscPipe 8.3, following my previous steps: makeDiscreteSkyMap.py => mosaic.py => coaddDriver.py. But mosaci.py is not found in hscPipe 8.3. I also can’t find the meas_mosaic in my installation and source code page. I wonder, is there a change or modification for coadding HSC images?
Cheers,
Charles
I don’t know the contents of hscPipe 8.3, but the Science Pipelines 19.0.0 release had this note: https://pipelines.lsst.io/releases/v19_0_0.html#removal-of-meas-mosaic
Here are the commands I ran for a recent reduction run with hscPipe 8.3:
singleFrameDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-G --job sfm-g --cores 120 --batch-type=slurm --mpiexec='-bind-to socket' --time 600
singleFrameDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-R^HSC-R2 --job sfm-r --cores 120 --batch-type=slurm --mpiexec='-bind-to socket' --time 600
singleFrameDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-I^HSC-I2 --job sfm-i --cores 120 --batch-type=slurm --mpiexec='-bind-to socket' --time 600
makeDiscreteSkyMap.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-G^HSC-R^HSC-R2^HSC-I^HSC-I2
jointcal.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-G tract=0 | tee jointcal-g.log &
jointcal.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-R^HSC-R2 tract=0 | tee jointcal-r.log &
jointcal.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-I^HSC-I2 tract=0 | tee jointcal-i.log &
fgcmMakeLut.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun
fgcmCalibrateTract.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-G^HSC-R^HSC-R2^HSC-I^HSC-I2 tract=0 |& tee fgcm.log
coaddDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id tract=0 filter=HSC-G --selectId field=INTERESTING filter=HSC-G --job coadd-g --time 300 --node 1 --procs 14 --batch-type=slurm --mpiexec='-bind-to socket' --batch-submit='--mem 128000'
coaddDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id tract=0 filter=HSC-G --selectId field=INTERESTING filter=HSC-R^HSC-R2 --job coadd-r --time 300 --node 1 --procs 14 --batch-type=slurm --mpiexec='-bind-to socket' --batch-submit='--mem 128000' -c makeCoaddTempExp.doApplySkyCorr=False
coaddDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id tract=0 filter=HSC-G --selectId field=INTERESTING filter=HSC-I^HSC-I2 --job coadd-i --time 300 --node 1 --procs 14 --batch-type=slurm --mpiexec='-bind-to socket' --batch-submit='--mem 128000' -c makeCoaddTempExp.doApplySkyCorr=False
multiBandDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id tract=0 filter=HSC-G^HSC-R^HSC-I --batch-type=slurm --nodes 10 --procs 14 --time 10000 --mpiexec='-bind-to socket' --batch-submit='--mem 128000' --job multiband
Note that I have explicitly disabled the sky correction, because for my particular science case, I don’t want it.
Hi @price
I encountered a message when I am running fgcmCalibrateTract.py (See following error message). Is the fgcm calibration necessary for coaddDriver.py. I didn’t use that when I coadd using 7.9.1.
fgcmCalibrateTract.fgcmBuildStars INFO: Spooling out 1041763 observation indices.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: Found version 0 reference catalog with old style units in schema.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: runmeas_algorithms/bin/convert_refcat_to_nJy.pyto convert fluxes to nJy.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: See RFC-575 for more details.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader INFO: Converted refcat flux fields to nJy (name, units): (g_flux, ‘’); (r_flux, ‘’); (i_flux, ‘’); (z_flux, ‘’); (y_flux, ‘’); (i_fluxSigma, ‘’); (y_fluxSigma, ‘’); (r_fluxSigma, ‘’); (z_fluxSigma, ‘’); (g_fluxSigma, ‘’)
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: Found version 0 reference catalog with old style units in schema.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: runmeas_algorithms/bin/convert_refcat_to_nJy.pyto convert fluxes to nJy.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: See RFC-575 for more details.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader INFO: Converted refcat flux fields to nJy (name, units): (g_flux, ‘’); (r_flux, ‘’); (i_flux, ‘’); (z_flux, ‘’); (y_flux, ‘’); (i_fluxSigma, ‘’); (y_fluxSigma, ‘’); (r_fluxSigma, ‘’); (z_fluxSigma, ‘’); (g_fluxSigma, ‘’)
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.referenceSelector INFO: Selected 289170/492043 references
fgcmCalibrateTract FATAL: Failed: too many indices for array
/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/meas_algorithms/8.0-hsc+1/python/lsst/meas/algorithms/sourceSelector.py:335: RuntimeWarning: divide by zero encountered in true_divide
signalToNoise = catalog[self.fluxField]/catalog[self.errField]
Traceback (most recent call last):
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcmcal/8.3-hsc/python/lsst/fgcmcal/fgcmCalibrateTract.py”, line 135, in call
results = task.runDataRef(butler, dataRefList)
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/8.0-hsc+1/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcmcal/8.3-hsc/python/lsst/fgcmcal/fgcmCalibrateTract.py”, line 267, in runDataRef
fgcmStarObservationCat)
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcmcal/8.3-hsc/python/lsst/fgcmcal/fgcmBuildStars.py”, line 875, in fgcmMatchStars
fgcmMakeStars.makeReferenceMatches(self.fgcmLoadReferenceCatalog)
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcm/8.3-hsc/lib/python/fgcm/fgcmMakeStars.py”, line 827, in makeReferenceMatches
self.starConfig[‘referenceFilterNames’])
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcmcal/8.3-hsc/python/lsst/fgcmcal/fgcmLoadReferenceCatalog.py”, line 244, in getFgcmReferenceStarsSkyCircle
fgcmRefCat[‘refMag’][:, :] = 99.0
IndexError: too many indices for array
Hi Charles, I noticed this request for help with an error message from fgcmCalibrateTract.py is over two weeks old, and I wanted to check in and see if a solution was found, or if this remains a problem?
No, I haven’t really solve this issue. I just skip fgcmMakeLut.py and fgcmCalibrateTract.py, then go to coaddDriver.py directly. Do you know what is the propose of fgcm?
I’m not very familiar with this kind of processing, no, so I can only point you to the documentation for it (FgcmCalibrateTractTask.html), which you probably already know about. Perhaps @price can chime in again on the necessity of fgcmCalibrateTract.py.
@ktl that link to the release notes returns a 404 not found error for me, is it a broken link?
Yes, the recent reorganization of pipelines.lsst.io caused the link to break. I have fixed it above.
I believe the IndexError from fgcmCalibrateTract.py comes from running it on a dataset containing observations in a single band, rather than multiple bands as required by fgcm. If that’s the case, you’ll need to use jointcal to calculate the photometric correction: add -c doPhotometry=True to the jointcal.py command-line. Then you’ll need to specify -c makeCoaddTempExp.externalPhotoCalibName=jointcal when running coaddDriver.py.
(Filed DM-26000 to have fgcm fail gracefully on single-band data.)
Yes, my process only includes a single band data. I will test what you suggested here. Thanks!!!