Will there be external TAP access to RSP DP0.2 tables?

[mwv (Michael Wood-Vasey)]

For DP0.2 will it be possible to access the RSP through TAP from an external site. E.g., directly running a TAP client (e.g., pyvo.dal.TAPService) from my laptop or NERSC?

[mwv (Michael Wood-Vasey)]

I am asking about the opposite direction of access from the following two posts, which were about accessing an external TAP service from the RSP:

Accessing External TAP Services from an RSP Jupyter Notebook

Informational: At today’s (20 May 2022) DP0 Delegates working meeting, there was a question on whether it was possible to access and query external TAP services from within an RSP Jupyter notebook. In particular, the question was whether it was possible to query the PanSTARRS1 TAP service from within an RSP Juptyer notebook from https://data.lsst.cloud. Fortunately, this was previously asked (and answered!) by Robert Nikutta in a Community Forum post in July 2021 (link. Basically, although …

How can I query external TAP services?

Hi team, how could one define an external TAP service URL, and query for data there? I was hoping that the get_tap_service() method from rubin_jupyter_utils.lab.notebook method would accept an argument to define a TAP URL, but it does not appear to be the case. Is there another way? Thank you, Robert

[DouglasLTucker]

For DP0.1, I was able to access the RSP TAP services using TOPCAT on my laptop; so I believe this will be the case. Let me revisit this question in full a little later today…

[mwv (Michael Wood-Vasey)]

I get

DALServiceError: 401 Client Error: Unauthorized for url: https://data.lsst.cloud/api/tap/sync

When I try to do it from external to the RSP.

[ktl (K-T Lim)]

You will need to provide a token in some way (authorization header or as username/password) to access TAP from outside the RSP. See SQR-039: Discussion of authentication and authorization for Science Platform.

As Douglas says, this works today (via https://data.lsst.cloud/auth/tokens/), but I’m not sure it’s as “pretty” as we would like, so there isn’t full documentation for it yet.

[mwv (Michael Wood-Vasey)]

Thanks for the pointer to that SQR document, @ktl, I’ll try it out.

@frossie Are there any performance limitations I should anticipate? Or, “How can I be a nice DP0 Delegate and not break things?”

[DouglasLTucker]

Sorry for the delay. I had to dig into the code a bit…

Anyway, @MelissaGraham pointed me to this useful piece Rubin DP0 Delegate documenation:
https://data.lsst.cloud/api-aspect .

Once you have your data.lsst.cloud TAP service token, you can either use it other TAP access services. I will give two cases: (1) via TOPCAT and (2) via a NOIRLAB DataLab Jupyter Notebook.

TOPCAT

1. Start up TOPCAT.
   (see http://www.star.bris.ac.uk/~mbt/topcat/ )

2. Click on 'Table Access Protocal (TAP) Query under the “VO” menu.
   

   

3. Fill in https://data.lsst.cloud/api/tap in the “TAP URL” window and click the “Use Service” button.
   

   

4. Fill in your security token under “User” in the Authentication window that pops up. Leave the “Password” blank. Click OK.
   

   

5. Now you have access to the RSP TAP service from TOPCAT.
   

   

NOIRLAB Data Lab

Here, basically you want to use the basic pyvo commands, not the specialized DataLab convenience functions.

1. Go to the NOIRLAB DataLab ( https://datalab.noirlab.edu/ ) and launch a Jupyter Notebook.

2. At the very minumun, import the pyvo and requests python modules:

import pyvo
import requests

3. Define the data.lsst.cloud TAP server URL and your security token:

tap_url = 'https://data.lsst.cloud/api/tap'
token = 'yy-xxxxxx'   # insert your own RSP TAP server security token here.

4. Set up appropriate authorization to access the RSP TAP server:

s = requests.Session()
s.headers["Authorization"] = "Bearer " + token
auth = pyvo.auth.authsession.AuthSession()
auth.credentials.set("lsst-token", s)
auth.add_security_method_for_url(tap_url, "lsst-token")
auth.add_security_method_for_url(tap_url + "/sync", "lsst-token")
auth.add_security_method_for_url(tap_url + "/async", "lsst-token")
auth.add_security_method_for_url(tap_url + "/tables", "lsst-token")

5. Access the RSP TAP servics:

tap = pyvo.dal.TAPService(tap_url, auth)

6. Run a query:

query = "SELECT * FROM tap_schema.schemas"
results = tap.run_sync(query)
results.to_table()

Below, I have screenshots of a full Jupyter notebook running the above in the NOIRLAB DataLab (well, with my security token changed to a generic yy-XXXXX):

---

I hope this helps! Apologies for the gory details, but I hope to be able to go back to this and remind myself how this works in the future(!).

Thanks!

[ktl (K-T Lim)]

Oh, it is “official”, then.

For notebook steps 4+5, I found it simpler to do:

cred = pyvo.auth.CredentialStore()
cred.set_password("x-oauth-basic", token)
tap = pyvo.dal.TAPService(tap_url, cred.get("ivo://ivoa.net/sso#BasicAA"))

(and no need to import requests)

[mwv (Michael Wood-Vasey)]

Thanks, all! I tried @ktl 's solution and it works perfectly.

[mwv (Michael Wood-Vasey)]

Thanks, @DouglasLTucker I appreciate the gory details for exactly the purpose of providing a good detailed answer for people searching this in the future (which will likely include us). I wasn’t familiar with using requests and appreciate learning about building the auth by specifying what to do for the different parts of the TAP service.

[gpdf (Gregory Dubois-Felsmann)]

@mwv I would prefer @ktl’s more concise recipe be propagated. It’s the result of some recent work on improving Python interfaces to authenticated IVOA services, and it’ll be more robust than enumerating specific endpoints (which could very well change over time - they are not required by the standard to keep those names).

[mwv (Michael Wood-Vasey)]

Yep. Here’s what I wrote up and circulated to DESC as a simple example test of NOIRLab and RSP TAP services.

test_rsp_tap_service.py (1.5 KB)