Gaia DR2 refcat on lsst-dev: gaia_dr2_20200414
The complete Gaia DR2 catalog is now available on lsst-dev at /datasets/refcats/htm/v1/gaia_dr2_20200414 (see RFC-634). This catalog contains about 1.7 x 10^9 sources between G magnitude ~7 - 21 in 131,073 HTM pixels (one file per pixel, shard level 7). The data totals about 310 GB. The directory contains a README.txt with more detailed information.
This is the first LSST refcat to contain coordinate errors, proper motions, and parallaxes. The coordinate errors will be incorporated into future runs of jointcal to improve its fitting and uncertainty estimates. The LSST Science Pipelines currently have some capacity for incorporating proper motion, but do not have any facility for using parallaxes: we plan to use this refcat to help develop such functionality.
Because of the large number of files, we do not recommend running ls or using tab-completion within this directory. Other than the HTM files, the directory contains: README.txt with some summary information; config.py used by LoadIndexedReferenceObjectsTask when reading the refcat; and IngestIndexedReferenceTask.py containing the configuration used to generate the refcat from the original data.
Using this refcat
The HSC datasets gen2 butler repository contains a ref_cats symlink to the HTM v1 reference catalogs, so no path updates are necessary to use this new refcat with HSC. Other lsst-dev /datasets may need an appropriate symlink created to make this refcat available to their respective butlers.
You can use this reference catalog with processCcd by specifying the following configuration file:
config.processCcd.calibrate.astromRefObjLoader.ref_dataset_name = "gaia_dr2_20200414"
config.processCcd.calibrate.astromRefObjLoader.anyFilterMapsToThis = "phot_g_mean"
or, with a LSST Science Pipelines version older than v21 (prior to anyFilterMapsToThis):
config.processCcd.calibrate.astromRefObjLoader.ref_dataset_name = "gaia_dr2_20200414"
config.processCcd.calibrate.astromRefObjLoader.filterMap = {}
for source in ('u', 'g', 'r', 'i', 'z', 'y'):
config.processCcd.calibrate.astromRefObjLoader.filterMap[source] = "phot_g_mean"
As of v21, jointcal's default configuration uses Gaia DR2, so no additional configuration is necessary. For Pipelines versions older than v21, use the following configuration to use this refcat with jointcal:
config.astrometryRefObjLoader.ref_dataset_name = 'gaia_DR2'
# This refcat includes coordinate errors, so we don't have to fake them!
config.astrometryReferenceErr = None
for source in ('u', 'g', 'r', 'i', 'z', 'y'):
config.astrometryRefObjLoader.filterMap[source] = "phot_g_mean"
Note that we do not have color terms for Gaia and the gaia magnitudes do not readily map to any LSST or HSC bands, so we only recommend its use for astrometric calibration, not photometric.
If you were previously using the Gaia DR1 or the PS1 refcats, you may have to alter your astrometric configuration to account for the much higher precision of this reference catalog. We do not yet have recommendations on what configuration changes will be necessary: a magnitude or S/N cut are a likely candidates. Please post below with questions or with new astrometric fitting configurations that you have success with.
Refcat READMEs
The lsst-dev:/datasets/refcats/htm directory now contains a README.txt with some summary information about the HTM reference catalogs. It summarizes the HTM refcat directory and notes which catalog directories have separate README.txt files with more detailed information (currently only v1/gaia_dr2_20200414/ and v1/ps1_pv3_3pi_20170110/).
Producers and maintainers of new LSST-style reference catalogs can refer to the developer guide datasets page for information about what to put in a refcat README.
