Accessing External TAP Services from an RSP Jupyter Notebook

[DouglasLTucker]

Informational:

At today’s (20 May 2022) DP0 Delegates working meeting, there was a question on whether it was possible to access and query external TAP services from within an RSP Jupyter notebook. In particular, the question was whether it was possible to query the PanSTARRS1 TAP service from within an RSP Juptyer notebook from https://data.lsst.cloud.

Fortunately, this was previously asked (and answered!) by Robert Nikutta in a Community Forum post in July 2021 (link.

Basically, although there is a convenience function, get_tap_service() within the rubin_jupyter_utils.lab.notebook module, it appears it only accesses TAP services internal to Rubin. (An example of using get_tap_service()for accessing the Rubin TAP service can be found in the excellent DP0 tutorial notebook, 02_Intermediate_TAP_Query.ipynb.)

Robert Nikutta discovered that get_tap_service() uses pyvo.dal.TAPService() “under the hood”; so, to access and query external TAP services, once can use the following:

import pyvo
tap = pyvo.dal.TAPService('https://datalab.noirlab.edu/tap')
tap.run_sync('select count(*) from gaia_edr3.gaia_source')

<Table length=1>
  COUNT   
  int32   
----------
1811709771

We confirmed that Robert’s solution still works at today’s meeting. Now, how to access the PanSTARRS1 TAP service from an RSP notebook on https://data.lsst.cloud? First, we need to find the access url for the PanSTARRS1 TAP service. There are several ways to go about this:

1. Google “TAP access endpoint panstarrs” in your browser. This will lead you to the following page among the top matches to your google search:
   

   
   
   You will want to click on the ‘+’ sign next to “Table Access Protocol” to find the URL you will need:
   

2. Instead of Googling, you can go straight to the STScI NAVO directory at http://vao.stsci.edu/keyword-search/ and enter “panstarrs” in the search box. It will return you several links (the top of which goes to the above webpage) for you to investigate:
   

   

3. If you are familiar with TOPCAT, you can use its TAP service functionality to find the correct URL; e.g.:
   

   

(In a future post, we will review how to access and query the Rubin TAP service from within TOPCAT.)

[DouglasLTucker]

In addition, one can search for access urls for external TAP servers within an RSP Jupyter notebook itself. Here’s how to do that for the PanSTARRS1 DR2 TAP server from an RSP Jupyter notebook running on https://data.lsst.cloud:

from pyvo.registry import search as regsearch
services = regsearch(keywords=['panstarrs'], servicetype='tap')

for item in services:
    print(item.res_title+':  '+item.access_url+'\n')
    print(item.res_description+'\n\n')

-----------------------

MAST STScI CAOM and ObsCore TAP service:  http://vao.stsci.edu/CAOMTAP/TapService.aspx

The MAST Archive at STScI TAP end point for observational data, saved in the Common Archive Data Model format and made available through the ObsCore limited view.  

The Table Access Protocol (TAP) lets you execute queries against our database tables, and inspect various metadata.  Upload is not currently supported.
A partial list of missions and projects with data available through the CAOMTAP service are: BEFS, EUVE, FUSE, GALEX, HLA, HST, HUT, IUE, K2, KEPLER, PS1 (PanSTARRS 1), SPITZER_SHA, SWIFT, TESS, TUES, WUPPE.
    
Tables exposed through this endpoint include the ObsCore (version 1.1) view of our CAOM (version 2.4) database, as well as all other CAOM tables


MAST STScI PanSTARRS 1 DR2 TAP service:  http://vao.stsci.edu/PS1DR2/TapService.aspx

The Panoramic Survey Telescope and Rapid Response System (Pan-STARRS or PS1) is a wide-field imaging facility developed at the University of Hawaii's Institute for Astronomy for a variety of scientific studies from the nearby to the very distant Universe. The PS1 catalog includes measurements in five filters (grizy) covering 30,000 square degrees of the sky north of declination -30 degrees, with typically ~12 epochs for each filter. This interface allows searches for the mean measurements and the deeper stacked measurements from images combining all the epochs. The DR2 release also includes the detection catalog containing all the multi-epoch observations. 

The Table Access Protocol (TAP) lets you execute queries against our database tables, and inspect various metadata.  Upload is not currently supported.

[DouglasLTucker]

(For info, see: Registry (pyvo.registry) — pyvo v1.3.dev732 .)

[gpdf (Gregory Dubois-Felsmann)]

While the OP did not ask about this, it’s worth pointing out to our DP0 users that the RSP Portal Aspect can also be used to access external TAP services that are not already in the provided list.