Mosaic.py not found in hscPipe 8.3

Hi,

I would like to stack HSC images which were processed by hscPipe 8.3, following my previous steps: makeDiscreteSkyMap.py => mosaic.py => coaddDriver.py. But mosaci.py is not found in hscPipe 8.3. I also can’t find the meas_mosaic in my installation and source code page. I wonder, is there a change or modification for coadding HSC images?

Cheers,
Charles

I don’t know the contents of hscPipe 8.3, but the Science Pipelines 19.0.0 release had this note: https://pipelines.lsst.io/releases/v19_0_0.html#removal-of-meas-mosaic

Here are the commands I ran for a recent reduction run with hscPipe 8.3:

singleFrameDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-G --job sfm-g --cores 120 --batch-type=slurm --mpiexec='-bind-to socket' --time 600
singleFrameDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-R^HSC-R2 --job sfm-r --cores 120 --batch-type=slurm --mpiexec='-bind-to socket' --time 600
singleFrameDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-I^HSC-I2 --job sfm-i --cores 120 --batch-type=slurm --mpiexec='-bind-to socket' --time 600

makeDiscreteSkyMap.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-G^HSC-R^HSC-R2^HSC-I^HSC-I2

jointcal.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-G tract=0 | tee jointcal-g.log &
jointcal.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-R^HSC-R2 tract=0 | tee jointcal-r.log &
jointcal.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-I^HSC-I2 tract=0 | tee jointcal-i.log &

fgcmMakeLut.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun
fgcmCalibrateTract.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id field=INTERESTING filter=HSC-G^HSC-R^HSC-R2^HSC-I^HSC-I2 tract=0 |& tee fgcm.log

coaddDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id tract=0 filter=HSC-G --selectId field=INTERESTING filter=HSC-G --job coadd-g --time 300 --node 1 --procs 14 --batch-type=slurm --mpiexec='-bind-to socket' --batch-submit='--mem 128000'
coaddDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id tract=0 filter=HSC-G --selectId field=INTERESTING filter=HSC-R^HSC-R2 --job coadd-r --time 300 --node 1 --procs 14 --batch-type=slurm --mpiexec='-bind-to socket' --batch-submit='--mem 128000' -c makeCoaddTempExp.doApplySkyCorr=False 
coaddDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id tract=0 filter=HSC-G --selectId field=INTERESTING filter=HSC-I^HSC-I2 --job coadd-i --time 300 --node 1 --procs 14 --batch-type=slurm --mpiexec='-bind-to socket' --batch-submit='--mem 128000' -c makeCoaddTempExp.doApplySkyCorr=False 

multiBandDriver.py /path/to/DATA --calib /path/to/CALIB --rerun myRerun --id tract=0 filter=HSC-G^HSC-R^HSC-I --batch-type=slurm --nodes 10 --procs 14 --time 10000 --mpiexec='-bind-to socket' --batch-submit='--mem 128000' --job multiband

Note that I have explicitly disabled the sky correction, because for my particular science case, I don’t want it.

Hi @price

I encountered a message when I am running fgcmCalibrateTract.py (See following error message). Is the fgcm calibration necessary for coaddDriver.py. I didn’t use that when I coadd using 7.9.1.

fgcmCalibrateTract.fgcmBuildStars INFO: Spooling out 1041763 observation indices.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: Found version 0 reference catalog with old style units in schema.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: run meas_algorithms/bin/convert_refcat_to_nJy.py to convert fluxes to nJy.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: See RFC-575 for more details.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader INFO: Converted refcat flux fields to nJy (name, units): (g_flux, ‘’); (r_flux, ‘’); (i_flux, ‘’); (z_flux, ‘’); (y_flux, ‘’); (i_fluxSigma, ‘’); (y_fluxSigma, ‘’); (r_fluxSigma, ‘’); (z_fluxSigma, ‘’); (g_fluxSigma, ‘’)
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: Found version 0 reference catalog with old style units in schema.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: run meas_algorithms/bin/convert_refcat_to_nJy.py to convert fluxes to nJy.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader WARN: See RFC-575 for more details.
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.refObjLoader INFO: Converted refcat flux fields to nJy (name, units): (g_flux, ‘’); (r_flux, ‘’); (i_flux, ‘’); (z_flux, ‘’); (y_flux, ‘’); (i_fluxSigma, ‘’); (y_fluxSigma, ‘’); (r_fluxSigma, ‘’); (z_fluxSigma, ‘’); (g_fluxSigma, ‘’)
fgcmCalibrateTract.fgcmBuildStars.fgcmLoadReferenceCatalog.referenceSelector INFO: Selected 289170/492043 references
fgcmCalibrateTract FATAL: Failed: too many indices for array
/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/meas_algorithms/8.0-hsc+1/python/lsst/meas/algorithms/sourceSelector.py:335: RuntimeWarning: divide by zero encountered in true_divide
signalToNoise = catalog[self.fluxField]/catalog[self.errField]
Traceback (most recent call last):
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcmcal/8.3-hsc/python/lsst/fgcmcal/fgcmCalibrateTract.py”, line 135, in call
results = task.runDataRef(butler, dataRefList)
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/8.0-hsc+1/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcmcal/8.3-hsc/python/lsst/fgcmcal/fgcmCalibrateTract.py”, line 267, in runDataRef
fgcmStarObservationCat)
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcmcal/8.3-hsc/python/lsst/fgcmcal/fgcmBuildStars.py”, line 875, in fgcmMatchStars
fgcmMakeStars.makeReferenceMatches(self.fgcmLoadReferenceCatalog)
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcm/8.3-hsc/lib/python/fgcm/fgcmMakeStars.py”, line 827, in makeReferenceMatches
self.starConfig[‘referenceFilterNames’])
File “/array/users/ytchen/hscPipe830_ichiro/stack/miniconda3-4.7.10-4d7b902/Linux64/fgcmcal/8.3-hsc/python/lsst/fgcmcal/fgcmLoadReferenceCatalog.py”, line 244, in getFgcmReferenceStarsSkyCircle
fgcmRefCat[‘refMag’][:, :] = 99.0
IndexError: too many indices for array

Hi Charles, I noticed this request for help with an error message from fgcmCalibrateTract.py is over two weeks old, and I wanted to check in and see if a solution was found, or if this remains a problem?

No, I haven’t really solve this issue. I just skip fgcmMakeLut.py and fgcmCalibrateTract.py, then go to coaddDriver.py directly. Do you know what is the propose of fgcm?

I’m not very familiar with this kind of processing, no, so I can only point you to the documentation for it (FgcmCalibrateTractTask.html), which you probably already know about. Perhaps @price can chime in again on the necessity of fgcmCalibrateTract.py.

@ktl that link to the release notes returns a 404 not found error for me, is it a broken link?

Yes, the recent reorganization of pipelines.lsst.io caused the link to break. I have fixed it above.

1 Like