Gaia DR2 reference catalog in LSST format

astrometry
refcat
calibration
Tags: #<Tag:0x00007f61a860bc68> #<Tag:0x00007f61a860ba60> #<Tag:0x00007f61a860b308>

(John Parejko) #1

Gaia DR2 refcat on lsst-dev: gaia_dr2_20190808

The full Gaia DR2 catalog is now available on lsst-dev at /datasets/refcats/htm/v1/gaia_dr2_20190808 (see RFC-634). This catalog contains about 1.7 x 10^9 sources between G magnitude 11.6 - 21.4 in 131,073 HTM pixels (one file per pixel, shard level 7). The data totals about 310 GB. The directory contains a README.txt with more detailed information.

This is the first LSST refcat to contain coordinate errors, proper motions, and parallaxes. The coordinate errors will be incorporated into future runs of jointcal to improve its fitting and uncertainty estimates. The LSST Science Pipelines currently have some capacity for incorporating proper motion, but do not have any facility for using parallaxes: we plan to use this refcat to help develop such functionality.

Because of the large number of files, we do not recommend running ls or using tab-completion within this directory. Other than the HTM files, the directory contains: README.txt with some summary information; config.py used by LoadIndexedReferenceObjectsTask when reading the refcat; and IngestIndexedReferenceTask.py containing the configuration used to generate the refcat from the original data.

Using this refcat

The HSC datasets gen2 butler repository contains a ref_cats symlink to the HTM v1 reference catalogs, so no path updates are necessary to use this new refcat with HSC. Other lsst-dev /datasets may need an appropriate symlink created to make this refcat available to their respective butlers.

You can use this reference catalog with HSC processCcd by specifying the following configuration file:

config.processCcd.calibrate.astromRefObjLoader.ref_dataset_name = "gaia_dr2_20190808"
config.processCcd.calibrate.astromRefObjLoader.filterMap = {}
for source in ('u', 'g', 'r', 'i', 'z', 'y'):
    config.processCcd.calibrate.astromRefObjLoader.filterMap[source] = "phot_g_mean_mag"

Similarly, to use this refcat with jointcal, use the following configuration:

config.astrometryRefObjLoader.ref_dataset_name = 'gaia_DR2'
# This refcat includes coordinate errors, so we don't have to fake them!
config.astrometryReferenceErr = None

for source in ('u', 'g', 'r', 'i', 'z', 'y'):
    config.astrometryRefObjLoader.filterMap[source] = "phot_g_mean_mag"

Note that we do not have color terms for Gaia and the gaia magnitudes do not readily map to any LSST or HSC bands, so we only recommend its use for astrometric calibration, not photometric.

If you were previously using the Gaia DR1 or the PS1 refcats, you may have to alter your astrometric configuration to account for the much higher precision of this reference catalog. We do not yet have recommendations on what configuration changes will be necessary: a magnitude or S/N cut are a likely candidates. Please post below with questions or with new astrometric fitting configurations that you have success with.

Refcat READMEs

The lsst-dev:/datasets/refcats/htm directory now contains a README.txt with some summary information about the HTM reference catalogs. It summarizes the HTM refcat directory and notes which catalog directories have separate README.txt files with more detailed information (currently only v1/gaia_dr2_20190808/ and v1/ps1_pv3_3pi_20170110/).

Producers and maintainers of new LSST-style reference catalogs can refer to the developer guide datasets page for information about what to put in a refcat README.


(John Parejko) #2

Warning: the reference epoch in the gaia dr2 reference catalog listed above–20190808–is incorrect. I did not see that Gaia specified the reference epoch in TCB instead of UTC, resulting in a difference of about a minute. This is unlikely to matter for our uses, but we might as well get it right.

I’m fixing it on DM-22027, and will post an announcement here when the updated catalog is available on lsst-dev.

As we are not using parallax or proper motion in production currently, this error has no effect on users of this refcat, so if you have been making use of Gaia dr2, please continue to do so without worry.

Hat tip to Siegfried Eggl for pointing this out to me.