Local access pyvo DALServiceError

Support Data Preview 0
#1

Hi, I am trying to access the data previews from my local machine (Apple M1, Sonoma) using these instructions: Introduction to the RSP API Aspect — Vera C. Rubin Observatory Documentation for Data Preview 0.2

Unfortunately I keep getting the DALServiceError below. My credentials are correct, as I am able to follow the tutorial for topcat on my machine and I can run the pyvo code when on the RSP. I was wondering if there was a version issue with pyvo, so I updated to pyvo=1.5.1 (I also tried creating a fresh conda environment for pyvo) but the error persists.

I saw similar forum topics but the main advice appeared to be log in and out, which I have done. Any advice would be greatly appreciated!

Traceback (most recent call last):
  File "/Users/jrobinson/miniconda3/envs/pyvo/lib/python3.12/site-packages/pyvo/dal/query.py", line 244, in execute_votable
    return votableparse(self.execute_stream(post=post).read)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jrobinson/miniconda3/envs/pyvo/lib/python3.12/site-packages/astropy/io/votable/table.py", line 164, in parse
    return tree.VOTableFile(config=config, pos=(1, 1)).parse(iterator, config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jrobinson/miniconda3/envs/pyvo/lib/python3.12/site-packages/astropy/io/votable/tree.py", line 4167, in parse
    vo_raise(E19, (), config, pos)
  File "/Users/jrobinson/miniconda3/envs/pyvo/lib/python3.12/site-packages/astropy/io/votable/exceptions.py", line 124, in vo_raise
    raise exception_class(args, config, pos)
astropy.io.votable.exceptions.E19: None:1:0: E19: File does not appear to be a VOTABLE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jrobinson/adler/notebooks/rsp_tap_pyvo.py", line 18, in <module>
    results = rsp_tap.run_sync(query)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jrobinson/miniconda3/envs/pyvo/lib/python3.12/site-packages/pyvo/dal/tap.py", line 278, in run_sync
    **keywords).execute()
                ^^^^^^^^^
  File "/Users/jrobinson/miniconda3/envs/pyvo/lib/python3.12/site-packages/pyvo/dal/tap.py", line 1115, in execute
    return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jrobinson/miniconda3/envs/pyvo/lib/python3.12/site-packages/pyvo/dal/query.py", line 246, in execute_votable
    self.raise_if_error()
  File "/Users/jrobinson/miniconda3/envs/pyvo/lib/python3.12/site-packages/pyvo/dal/query.py", line 255, in raise_if_error
    raise DALServiceError.from_except(e, self.queryurl)
pyvo.dal.exceptions.DALServiceError: 401 Client Error: Unauthorized for url: https://data.lsst.cloud/api/tap/sync/bgkfidcxvge019r9/run
#2

Since you are able to run RSP queries with TOPCAT and with pyvo on the RSP, I am wondering if there could be a problem with your ~/.rsp-tap.token file on your local machine. I don’t think it is a pyvo version issue, since I can run the query in Introduction to the RSP API Aspect — Vera C. Rubin Observatory Documentation for Data Preview 0.2 on my local machine using a pretty old version of pyvo (pyvo v1.4.2).

Maybe you could delete the current ~/.rsp-tap.token file on your local machine and re-create it using Step 1 of Introduction to the RSP API Aspect — Vera C. Rubin Observatory Documentation for Data Preview 0.2 ? You can use the same token as you used for TOPCAT (and, for that matter, what you used when you used pyvo on the RSP), as long as that token is not expired.

My big concern with the current ~/.rsp-tap.token file on your local machine is that perhaps the contents are corrupted in some way or that there is no “end of line” character after the token. In the latter case, listing the file might show something like this:

dtucker@mac-137051 ~ % cat .rsp-tap.token_fake
**-*************************************************%

instead of something like this:

dtucker@mac-137051 ~ % cat .rsp-tap.token_fake
**-*************************************************
dtucker@mac-137051 ~ %

(where “**-*************************************************” would be the token).

So maybe try recreating ~/.rsp-tap.token on your local machine.

I hope this helps!

#3

Hi Douglas, thanks for the advice. I had originally followed those instructions for creating the ~/.rsp-tap.token file but I made the file again just to be sure. I can confirm that I included the EOF and the output of cat ~/.rsp-tap.token looks like your second example. Unfortunately I’m getting the same error!

#4

I wonder if this is a mac thing (specifically a my mac thing…). I just tried the pyvo code (version 1.3) with my token on our department’s unix server (Ubuntu 22.04.4 LTS) and it works.

#5

Hi James, sorry for the problems you are having! Right now, I am grabbing at straws, but I am wondering if it could be a pyvo incompatibility with the M1 architecture of you Mac. I “googled” around and haven’t found anything, but I recall that some other software packages have yet to catch up with the newer Mac chip architecture. Let me ask around and see if I can find out…
Thanks!

1 Like
#6

Okay to recap, same user, same token, works from computer A (random linux box) but not computer B (the M1 mac). Yeah this is unlikely to be on our end however I am curious as to what is going on. @jrob93 - if you wouldn’t mind, can you please post a minimal code example that demonstrates this problem ?

1 Like
#7

Thanks Douglas, yeah I’ve encountered M1 related problems before so it wouldn’t surprise me if this could be the case!

#8

Hi, yes that recap is correct, what I am trying to do appears to work everywhere except on my M1 Macbook Pro. Here is the exact python script that I am running:

import pyvo
import os

RSP_TAP_SERVICE = 'https://data.lsst.cloud/api/tap'
homedir = os.path.expanduser('~')
token_file = os.path.join(homedir,'.rsp-tap.token')
with open(token_file, 'r') as f:
    token_str = f.readline()

cred = pyvo.auth.CredentialStore()
cred.set_password("x-oauth-basic", token_str)
credential = cred.get("ivo://ivoa.net/sso#BasicAA")
rsp_tap = pyvo.dal.TAPService(RSP_TAP_SERVICE, credential)

query = "SELECT * FROM tap_schema.schemas"
results = rsp_tap.run_sync(query)
print(results.to_table())

It’s just the same code given in the example documentation. Thanks for taking a look, please let me know if there is any other info I can send.

#9 
results = rsp_tap.run_sync(query)
print(results.to_table())

And the failure is on the run_sync(), right, not on the next line?

#10

Yup the failure is on this line:
results = rsp_tap.run_sync(query)

I think their error log in the first post confirms this.

FWIW all I can offer here is to confirm that code snippet works fine for me, from the NOIRLab Astro DataLab, with pyvo version 1.4.1.

#11

Sorry to dash hopes of an explanation, but the reproducible snippet does work fine on my M1.

James, sorry to ask again, but when you said it worked from a different machine, did you copy the exact token file you use on your M1 ? Or recreate it in some other way (cut/paste etc)

Also can you list the unix permissions of your token file
ls -al ~/.rsp-tap.token

#12

Hey, when I tried it on a different machine I repeated the steps in the tutorial to create the file securely and I copy-pasted the token into the new file at the prompt.

And the permissions are:

$ ls -al ~/.rsp-tap.token
-rw-------  1 jrobinson  staff  49 24 Apr 09:57 /Users/jrobinson/.rsp-tap.token

But I’m pretty sure that my token and file are fine, I’ve even experimented with temporarily pasting the token directly into the python script and the results are the same.

@frossie could you possibly share details on the conda environment (or similar) that works for you and I’ll try recreate it?

#13

Hi @frossie, just following up with you on @jrob93’s request for details on the conda environment you used to get things to work. Or @jrob93, perhaps you were able to figure out what the issue was?

#14

Hi - I am sorry I don’t use conda. My steps were

  • A fresh virtual env (python-venv, Python 3.11.5)
  • Get token and put in ~/.rsp-tap.token
  • Pip install pyvo
  • Run the code provided by James (which worked fine)
1 Like
#15

Hey @ryanlau I haven’t figured it out yet. I was thinking it was maybe something to do with conda so I was trying to make sure everything was up to date. Somewhat unrelated, I ended up having to do a fresh install of my base environment following these instructions (conda 24.1.1 libmamba solver not recognized · Issue #13353 · ContinuumIO/anaconda-issues · GitHub). I then made a fresh environment for pyvo to try and match @frossie:
conda create -n pyvo python=3.11.5
pip install pyvo

This resulted in the same error message. I also tried installing pyvo with conda (conda install astropy::pyvo and conda install astropy::pyvo) with the same result.

#16

Hi @jrob93! Just checking to see if this issue has been resolved. It certainly has been quite a head-scratcher! Thanks!

#17

Hi Douglas, thanks for checking up. Not resolved yet but I think the problem has been narrowed done to some sort of pyvo bug. We’ve ruled out token and network issues, and I’ve been running some tests on my mac and linux machines. I’ve attached the terminal output from these tests. Interestingly on my mac the pyvo run_sync/run_async fails as before, however in test rsp_stelios2.py I am able to retrieve the metadata tables okay. I’m not familiar with how pyvo works but it seems that the connection and authentication is going through but there is something within run_sync/run_async that is failing and that the error
pyvo.dal.exceptions.DALServiceError: 401 Client Error: Unauthorized for url
might be something of a red herring. I’ve been meaning to sit down and try print some output from within pyvo for diagnosis, will update again when I try that!
pyvo_tests_linux.txt (6.2 KB)
pyvo_tests_mac.txt (5.8 KB)