Pan-STARRS reference catalog in LSST format

The Pan-STARRS 3pi stack data (images and catalogs) were released to the world as DR1 on 2016 December 19. This provides a high-quality network of calibration sources over three quarters of the sky to a modest depth. Telescopes both large and small (depths overlapping i ~ 17 to 21 mag) observing fields north of Dec=-30 deg in the optical will have good PS1 calibration sources in the field, observed through the same column as the target sources, allowing simple relative calibration for astrometry, photometry and spectroscopy.

To allow us (and others) to take advantage of this, I have produced a point source catalog in the LSST reference catalog format and also in the astrometry.net format we have used in the past. Details on the contents, construction and use of the catalog are available from the README.txt file included in the catalog, which I will summarise here.

Overview

This reference catalog contains astrometry and grizy photometry for 2,990,470,528 point sources at Dec > -30 deg to i ~ 22.5 mag.

Users of this catalog should include the following acknowledgement in publications:

The Pan-STARRS1 Surveys (PS1) have been made possible through contributions of the Institute for Astronomy, the University of Hawaii, the Pan-STARRS Project Office, the Max-Planck Society and its participating institutes, the Max Planck Institute for Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial Physics, Garching, The Johns Hopkins University, Durham University, the University of Edinburgh, Queen’s University Belfast, the Harvard-Smithsonian Center for Astrophysics, the Las Cumbres Observatory Global Telescope Network Incorporated, the National Central University of Taiwan, the Space Telescope Science Institute, the National Aeronautics and Space Administration under Grant No. NNX08AR22G issued through the Planetary Science Division of the NASA Science Mission Directorate, the National Science Foundation under Grant No. AST-1238877, the University of Maryland, and Eotvos Lorand University (ELTE), the Los Alamos National Laboratory, and the Gordon and Betty Moore Foundation.

Relevant papers for information and citation include:

LSST-style format

The LSST-style format uses FITS files divided by HTM. You can download the LSST-style catalog here. Be warned that it’s rather large: 423 GB. You can download a subset of the catalog by grabbing the appropriate FITS files; see the section in the README.txt entitled “Field selections” for details.

You need to install it in your data repo:

ln -s /path/to/ps1_pv3_3pi_20170110/ /path/to/DATA_REPO/ref_cats

Then you need to adjust your configuration. Here’s a configuration for processCcd.py (either to put in your obs package as config/processCcd.py, or to specify on the command-line with --configfile). See also the section below on “Common configuration” for colorterms and filterMap.

from lsst.meas.algorithms import LoadIndexedReferenceObjectsTask
config.calibrate.astromRefObjLoader.retarget(LoadIndexedReferenceObjectsTask)
config.calibrate.astromRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110"
config.calibrate.photoRefObjLoader.retarget(LoadIndexedReferenceObjectsTask)
config.calibrate.photoRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110"
config.calibrate.photoCal.photoCatName = "ps1_pv3_3pi_20170110"

Astrometry.net-style format

The astrometry.net-style format uses custom FITS files divided by HEALPix and indexed using tools provided by astrometry.net. You can download the astrometry.net-style catalog here. This format is a bit smaller than the LSST-style format (it does not include as much detail, e.g., proper motions), but is still quite hefty at 396 GB. If you want a subset of the catalog, you will need to figure out the appropriate HEALPix using the get-healpix tool from astrometry.net (I’m not entirely sure whether it’s the “XY”, “RING” or “NESTED” scheme you want) and grab the *_and_<healpix>_[012].fits files. Or you can grab the appropriate files from the LSST-style format and generate your own astrometry.net-style catalog using the included scripts; see the README.txt under “Astrometry.net format” for details.

The downloaded catalog doesn’t need to be installed in the data repo, but it should be declared to eups:

eups declare astrometry_net_data ps1_pv3_3pi_20170110-and -r /path/to/ps1_pv3_3pi_20170110-and

Then, to use it, you need to:

setup -j astrometry_net_data ps1_pv3_3pi_20170110-and

No configuration changes are currently required, because astrometry.net-style catalogs are currently the default (this will likely change soon), but do see the section below on “Common configuration” for colorterms and filterMap.

Common configuration

Regardless of which style of catalog you use, you need to set the colorterms and filterMap appropriately for your camera. Here is an example configuration for processCcd.py:

# These colorterms are for HSC, included as an example
colorterms = config.calibrate.photoCal.colorterms
from lsst.pipe.tasks.colorterms import ColortermDict, Colorterm
colorterms.data["ps1*"] = ColortermDict(data={
    'g': Colorterm(primary="g", secondary="r", c0=0.00730066, c1=0.06508481, c2=-0.01510570),
    'r': Colorterm(primary="r", secondary="i", c0=0.00279757, c1=0.02093734, c2=-0.01877566),
    'r2': Colorterm(primary="r", secondary="i", c0=0.00117690, c1=0.00003996, c2=-0.01667794),
    'i': Colorterm(primary="i", secondary="z", c0=0.00166891, c1=-0.13944659, c2=-0.03034094),
    'i2': Colorterm(primary="i", secondary="z", c0=0.00180361, c1=-0.18483562, c2=-0.02675511),
    'z': Colorterm(primary="z", secondary="y", c0=-0.00907517, c1=-0.28840221, c2=-0.00316369),
    'y': Colorterm(primary="y", secondary="z", c0=-0.00156858, c1=0.14747401, c2=0.02880125),
})
# For the HSC r2 and i2 filters, use the r and i values from the catalog
for refObjLoader in (config.calibrate.astromRefObjLoader,
                     config.calibrate.photoRefObjLoader,
                     config.charImage.refObjLoader,
                     ):
    refObjLoader.filterMap['r2'] = 'r'
    refObjLoader.filterMap['i2'] = 'i'

EDIT Aug 2020: NCSA/LSST doesn’t seem to be running their web server at the moment. If you can’t access it, there’s a copy of the catalog available from http://tigress-web.princeton.edu/~pprice/ps1_pv3_3pi_20170110/ .

6 Likes

Rick White (STScI) sent around this summary of the PS1 archive usage since the data release:

I thought you would be interested to hear about the usage of the PS1 public archive at STScI. It has been 1 month since the PS1 data release on December 19. In that time:

15.4 TB of images have been downloaded to 15,500 different IP addresses.

3.2 million image cutouts (“postage stamps”) have been extracted.

2.5 million Virtual Observatory catalog queries have been done by 70 different IP addresses, returning information on 580 million objects.

644,000 database queries have been executed through our CasJobs interface by 99 different users, returning more than 7.6 billion rows of data. The total data volume downloaded from CasJobs is 4.4 TB.

16,100 catalog searches have been done using the MAST web search form.

For comparison, the image data volume in the current public release is about 100 TB, and the catalog includes approximately 10 billion objects in a 20 TB database.

Despite the heavy usage, I believe there have been no negative impacts on other STScI archive usage or on our internet access. Our IT infrastructure has shown a lot of resiliency in handling the load with little difficulty.

This was only possible through the efforts of the PS1 team, and the huge popularity of the data is a testament to the value of the survey!

I realise that this is a fairly old thread now, but I thought it may be the best place to put this for interested parties to find…

We’ve been using the Pan-STARRS catalogue for about a year now, and it’s producing excellent results (thanks @price). However, because it is the whole PS1 catalogue and because we require the full survey coverage, we’ve found it a bit “non-portable”; indeed, I’ve always refrained from putting it on my laptop due to its size.

Since we don’t need such a deep reference catalogue for our purposes, we’ve created a smaller LSST-format catalogue by extracting only sources brighter than 19th mag (g-band) which, after gzip compression, is “only” 29GB (it’s roughly double that after decompression).

If a shallower reference is suitable for your purposes, and you wish to use this smaller catalogue, you can grab it from:
http://astro.shef.ac.uk/lsstps1/

To help prevent bot-downloads, we’ve put it behind a simple login:
username: lsstps1
password: UoSlsst

Download it, unzip and untar it, and use it in the same way @price suggests in his earlier post. While we’ve done some (limited) testing and found it reproduces the results of the larger catalogue exactly, we haven’t tested it to exhaustion, and so accept no responsibility/liability etc.

Clearly, if you use this smaller catalogue then the same acknowledgements apply as in @price’s original post. Thanks again to @price for the far greater effort of generating and making the original LSST-format catalogue freely available in the first place.

4 Likes

As above, I realize that this is a fairly old thread, but I wanted to post this question here in case it’s useful to anyone else. I’m trying to follow these instructions in a gen3 world, and I have a few questions about setting up my ref cat for use by a small group of users on a local machine.

Specifically, I’m not sure of the best way to ingest the ps1_pv3_3pi_20170110 catalogue and have it referenced as a collection. I have the files linked above downloaded, and have them sym-linked into my butler repository, but I’m unsure of the best way to register these data as a ref cat collection. Are there any gen3 tools that will help set this up?

1 Like

We have no gen3 refcat ingest tools; refcats have to be converted from a gen2 repo.

Follow DM-29543 for upates.

1 Like

Hi! I want to share that I am trying to download the content from the catalog available, but not entirely. Taking a look at the README.txt from testdata_ci_hsc/ps1_pv3_3pi_20170110 of the tutorial of Getting Started with the LSST Science Pipelines It proposed a method of selecting fields:

Field selections
----------------

The 423 GB size of this catalog may be prohibitive for individual use.  In that
case, it's possible to copy individual files required for particular regions of
interest. Just grab the <shard>.fits files (plus config.py and this README.txt
file), where the list of shards is calculated from this code:

    from lsst.meas.algorithms.htmIndexer import HtmIndexer
    from lsst.afw.coord import IcrsCoord
    from lsst.afw.geom import degrees

    def getShards(ra, dec, radius):
        htm = HtmIndexer(depth=7)
        shards, onBoundary = htm.get_pixel_ids(IcrsCoord(ra*degrees, dec*degrees), radius*degrees)
        return shards

Nowadays I checked that the attributes htm.get_pixel_ids and IcrsCoord are no longer part if the lsst pipeline software, so I was able to accommodate the getShards function to the current attributes available:

from lsst.meas.algorithms.htmIndexer import HtmIndexer
import lsst.geom as geom

def getShards(lon,lat, radius):
    # lon goes from 0 to 2pi
    htm = HtmIndexer(depth=7)
    shards, onBoundary = htm.getShardIds(geom.SpherePoint(lon*geom.degrees, lat*geom.degrees), radius*geom.degrees)
    return shards

when I print the results of getShards(3,0,1) I get:

[131108 131111 253976 253979 131080 131081 131083 131109 131110 131116
131118 131119 131124 131126 131127 253956 253958 253959 253977 253978
253980 253981 253983 254008 254009 254011]

It looks like it’s working nice! Happy to receive any comments :slight_smile:

2 Likes

Well done, and thank you!

1 Like

Yesterday we released the Gen3 version of this tutorial and it can be found at

https://pipelines.lsst.io/v/weekly/getting-started/index.html

1 Like

I’ve generated the gen3 refcat ingest index .ecsv files on lsst-devl. See this post for more details: Gaia DR2 reference catalog in LSST format - #18 by parejkoj

@price – I noticed that this set of reference catalogs includes shards covering nearly all of the southern sky, even below the Dec ~ -30 deg cutoff of Pan-STARRS. Do you know what survey(s) the sources in these southern reference catalogs come from? For instance:

https://tigress-web.princeton.edu/~pprice/ps1_pv3_3pi_20170110/168357.fits

Has a few thousand objects, with Dec ~ -77 deg. Thanks very much…

It’s been a while since I generated that catalog, and even longer since I was involved with Pan-STARRS. And the cluster serving the data is down right now so I can’t look at that example. My guess is that it’s Gaia, which was used to seed the catalog and has somehow slipped through, but I don’t know if that’s right or even makes sense.