I am running hscPipe v5 beta4 on CentOS7
I mapped a rather large (8x8deg) region of the sky with HSC and would like to create a single stacked mosaic.
I successfully processed the individual images with singleFrameDriver.py.
I then ran makeDiscreteSkyMap.py and mosaic.py, but encounter two problems when running coaddDriver.py on the whole thing:
at some point it complains that too many files are open… it might be an OS issue?
it complains that some files are rejected "because size scatter is too large”. I suspect it’s because the TAN projection is not good enough for such large areas. I also suspect I should use makeSkymap.py instead of makeDiscreteSkyMap.py?
Can anyone tell me whether it is possible to make such a large image? If it is, could you help me with the detailed steps and commands until from the skymap to the assembleCoadd.py?
it complains that some files are rejected "because size scatter is too large”. I suspect it’s because the TAN projection is not good enough for such large areas. I also suspect I should use makeSkymap.py instead of makeDiscreteSkyMap.py?
This message is referring to the quality of the PSF. By default we reject images where the PSF modeling is not good. If you are not too concerned about the accuracy of your PSF, you can set the threshold to a large number:
For those unaware, hscPipe 5.x is based on LSST 13.0 (it actually branched a little later, but before the big pybind change).
A single FITS image covering 64 square degrees probably wouldn’t work, so our pipeline divides the sky up into “tracts” which are essentially defined by their projection center, plate scale, and projection scheme. Each tract is divided into “patches”. Each patch within a tract shares the Wcs of the tract, but have different integer offsets, with overlaps so we don’t run into edge issues. We have different schemes for generating tracts (tessellating the sphere); in my opinion, the two most useful are:
rings: the sphere is divided up into bands of declination, and tracts lie along each band. This is used by the HSC Survey, and is therefore the default for HSC data.
discrete: tracts are placed at positions nominated by the user (perhaps through the data, if you use makeDiscreteSkyMap.py).
You’ll need to decide which is suitable for your science. While discrete would allow you to create the “single stacked mosaic” you desire (using a single tract to cover your survey area), you’ll need to decide whether the distortion at the edges is tolerable. If you want to use rings, you run makeSkyMap.py (rings is the default for HSC); if you want to use discrete then it’s easiest to run makeDiscreteSkyMap.py (that sets the projection center and size based on the data; if you’re interested, there’s an alternative means to set those explicitly to values you choose). Then it’s just a matter of running coaddDriver.py to create the coadds. Don’t worry about assembleCoadd.py — that runs just a single element of what coaddDriver.py will run.
Now, your specific problems:
I haven’t seen an error like that in a long time, and never in this context. You might check your shell limits on the number of open file descriptors (ulimit -n). Can you provide some more details, like the command-line you’re running?
I believe the “size scatter” refers to the PSF ellipticity residuals. Our reductions of HSC images are aimed at producing good data for weak lensing studies, so images for which we have trouble modeling the PSF are not included in the coadd. You can adjust the threshold as Bob has indicated (there are ways to turn that behaviour off, but it’s more complicated and probably unnecessary).