Gaia DR2 reference catalog in LSST format

Tags: #<Tag:0x00007f61a4457600> #<Tag:0x00007f61a4457308> #<Tag:0x00007f61a4457088>

(John Parejko) #1

Gaia DR2 refcat on lsst-dev: gaia_dr2_20200414

The complete Gaia DR2 catalog is now available on lsst-dev at /datasets/refcats/htm/v1/gaia_dr2_20200414 (see RFC-634). This catalog contains about 1.7 x 10^9 sources between G magnitude ~7 - 21 in 131,073 HTM pixels (one file per pixel, shard level 7). The data totals about 310 GB. The directory contains a README.txt with more detailed information.

This is the first LSST refcat to contain coordinate errors, proper motions, and parallaxes. The coordinate errors will be incorporated into future runs of jointcal to improve its fitting and uncertainty estimates. The LSST Science Pipelines currently have some capacity for incorporating proper motion, but do not have any facility for using parallaxes: we plan to use this refcat to help develop such functionality.

Because of the large number of files, we do not recommend running ls or using tab-completion within this directory. Other than the HTM files, the directory contains: README.txt with some summary information; used by LoadIndexedReferenceObjectsTask when reading the refcat; and containing the configuration used to generate the refcat from the original data.

Using this refcat

The HSC datasets gen2 butler repository contains a ref_cats symlink to the HTM v1 reference catalogs, so no path updates are necessary to use this new refcat with HSC. Other lsst-dev /datasets may need an appropriate symlink created to make this refcat available to their respective butlers.

You can use this reference catalog with HSC processCcd by specifying the following configuration file:

config.processCcd.calibrate.astromRefObjLoader.ref_dataset_name = "gaia_dr2_20200414"
config.processCcd.calibrate.astromRefObjLoader.filterMap = {}
for source in ('u', 'g', 'r', 'i', 'z', 'y'):
    config.processCcd.calibrate.astromRefObjLoader.filterMap[source] = "phot_g_mean"

Similarly, to use this refcat with jointcal, use the following configuration:

config.astrometryRefObjLoader.ref_dataset_name = 'gaia_DR2'
# This refcat includes coordinate errors, so we don't have to fake them!
config.astrometryReferenceErr = None

for source in ('u', 'g', 'r', 'i', 'z', 'y'):
    config.astrometryRefObjLoader.filterMap[source] = "phot_g_mean"

Note that we do not have color terms for Gaia and the gaia magnitudes do not readily map to any LSST or HSC bands, so we only recommend its use for astrometric calibration, not photometric.

If you were previously using the Gaia DR1 or the PS1 refcats, you may have to alter your astrometric configuration to account for the much higher precision of this reference catalog. We do not yet have recommendations on what configuration changes will be necessary: a magnitude or S/N cut are a likely candidates. Please post below with questions or with new astrometric fitting configurations that you have success with.

Refcat READMEs

The lsst-dev:/datasets/refcats/htm directory now contains a README.txt with some summary information about the HTM reference catalogs. It summarizes the HTM refcat directory and notes which catalog directories have separate README.txt files with more detailed information (currently only v1/gaia_dr2_20200414/ and v1/ps1_pv3_3pi_20170110/).

Producers and maintainers of new LSST-style reference catalogs can refer to the developer guide datasets page for information about what to put in a refcat README.

(John Parejko) #2

Warning: the reference epoch in the gaia dr2 reference catalog listed above–20190808–is incorrect. I did not see that Gaia specified the reference epoch in TCB instead of UTC, resulting in a difference of about a minute. This is unlikely to matter for our uses, but we might as well get it right.

I’m fixing it on DM-22027, and will post an announcement here when the updated catalog is available on lsst-dev.

As we are not using parallax or proper motion in production currently, this error has no effect on users of this refcat, so if you have been making use of Gaia dr2, please continue to do so without worry.

Hat tip to Siegfried Eggl for pointing this out to me.

(John Parejko) #3

The updated Gaia DR2 refcat with the corrected epochs is now in place as gaia_dr2_20191105. I’ve updated the above post to refer to it instead of the 20190808 version, and also updated the respective readmes in the /datasets/refcats/htm directories.

(John Parejko) #4

I have discovered that the above Gaia DR2 reference catalogs have incorrect coordinate error fields (coord_raErr, coord_decErr) due to a unit conversion error. This primarily affects jointcal, as our AstrometryTask does not use the reference catalog errors when fitting single frame astrometry. If you have another use of this reference catalog that incorporates the coordinate errors, please update immediately.

I have corrected this as part of DM-24472, and regenerated the LSST reference catalog. The new catalog is now available on lsst-dev as gaia_dr2_20200414 on lsst-dev at /datasets/refcats/htm/v1. I have updated the above post to refer to it instead.