TAP Service Authorization

babel · September 18, 2022, 6:01pm

Today I tried to run the same code as yesterday, but I got
DALServiceError: 401 Client Error: Unauthorized for url: https://data.lsst.cloud/api/tap/sync

I have:

Get an instance of the TAP service

service = get_tap_service()
assert service is not None

assert service.baseurl == “https://data.lsst.cloud/api/tap”

and using the same query as before:

For more detailed analysis of results, converting

to a pandas dataframe is often very useful (“df” stands for “dataframe”)

df = service.search(query, maxrec=max_rec).to_table().to_pandas()

Use an assertion to make sure we got the correct number of results.

assert len(df) == 15670

len(df.coord_ra)

which gives me:

E19 Traceback (most recent call last)
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:242, in DALQuery.execute_votable(self, post)
241 try:
→ 242 return votableparse(self.execute_stream(post=post).read)
243 except Exception as e:

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/utils/decorators.py:546, in deprecated_renamed_argument..decorator..wrapper(*args, **kwargs)
544 warnings.warn(msg, warning_type, stacklevel=2)
→ 546 return function(*args, **kwargs)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/table.py:160, in parse(source, columns, invalid, verify, chunk_size, table_number, table_id, filename, unit_format, datatype_mapping, _debug_python_based_parser)
156 with iterparser.get_xml_iterator(
157 source,
158 _debug_python_based_parser=_debug_python_based_parser) as iterator:
159 return tree.VOTableFile(
→ 160 config=config, pos=(1, 1)).parse(iterator, config)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/tree.py:3601, in VOTableFile.parse(self, iterator, config)
3600 else:
→ 3601 vo_raise(E19, (), config, pos)
3602 config.update(self._get_version_checks())

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/exceptions.py:112, in vo_raise(exception_class, args, config, pos)
111 config = {}
→ 112 raise exception_class(args, config, pos)

E19: None:1:0: E19: File does not appear to be a VOTABLE

During handling of the above exception, another exception occurred:

DALServiceError Traceback (most recent call last)
Input In [5], in <cell line: 3>()
1 # For more detailed analysis of results, converting
2 # to a pandas dataframe is often very useful (“df” stands for “dataframe”)
----> 3 df = service.search(query, maxrec=max_rec).to_table().to_pandas()
5 # Use an assertion to make sure we got the correct number of results.
6 # assert len(df) == 15670
7 len(df.coord_ra)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/tap.py:248, in TAPService.run_sync(self, query, language, maxrec, uploads, **keywords)
219 def run_sync(
220 self, query, language=“ADQL”, maxrec=None, uploads=None,
221 **keywords):
222 “”"
223 runs sync query and returns its result
224
(…)
244 TAPResults
245 “”"
246 return self.create_query(
247 query, language=language, maxrec=maxrec, uploads=uploads,
→ 248 **keywords).execute()

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/tap.py:942, in TAPQuery.execute(self)
928 def execute(self):
929 “”"
930 submit the query and return the results as a TAPResults instance
931
(…)
940 for errors parsing the VOTable response
941 “”"
→ 942 return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:244, in DALQuery.execute_votable(self, post)
242 return votableparse(self.execute_stream(post=post).read)
243 except Exception as e:
→ 244 self.raise_if_error()
245 raise DALFormatError(e, self.queryurl)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:253, in DALQuery.raise_if_error(self)
251 if self._ex:
252 e = self._ex
→ 253 raise DALServiceError.from_except(e, self.queryurl)

DALServiceError: 401 Client Error: Unauthorized for url: https://data.lsst.cloud/api/tap/sync

DouglasLTucker · September 19, 2022, 3:29am

Dear Bob,
Hmmm… interesting. I wonder if the Rubin TAP service was having issues today.
Could you send me the query you were trying to run? You can either post it here or send it to me via a direct message on Slack.
Thanks!
Best regards,
Douglas

babel · September 19, 2022, 9:04pm

Hi Douglas, thank you for your response! I’m still getting the same error today. Here’s what I have:

Get an instance of the TAP service

service = get_tap_service()
assert service is not None

coord = SkyCoord(ra=62.0u.degree, dec=-37.0u.degree, frame=‘icrs’)
radius = 0.1 * u.deg

max_rec = 1000

query = "SELECT objectId, coord_ra, coord_dec, u_cModelFlux, g_cModelFlux, " \

"r_cModelFlux, i_cModelFlux, z_cModelFlux, y_cModelFlux, " \

"tract, patch, detect_isPrimary, refExtendedness " \

"FROM dp02_dc2_catalogs.Object " \

"WHERE CONTAINS(POINT(‘ICRS’, coord_ra, coord_dec),CIRCLE(‘ICRS’, " \

+ str(coord.ra.value) + ", " + str(coord.dec.value) + ", " \

+ str(radius.value) + " )) = 1 " \

“AND refExtendedness = 1 AND detect_isPrimary = 1”

query = "SELECT objectId, coord_ra, coord_dec, g_hsmShapeRegauss_e1, g_hsmShapeRegauss_e2, "
"u_ixx, r_ixx, i_ixx, z_ixx, y_ixx, refBand, "
"g_hsmShapeRegauss_flag, g_ixx, g_ixxDebiasedPSF, g_ixxPSF, g_ixxRound, g_ixy, "
"g_ixyDebiasedPSF, g_ixyPSF, g_ixyRound, g_iyy, g_iyyDebiasedPSF, "
"g_iyyPSF, g_iyyRound, shape_flag, shape_xx, shape_xy, shape_yy, "
"tract, patch, detect_isPrimary, refExtendedness "
"FROM dp02_dc2_catalogs.Object "
"WHERE CONTAINS(POINT(‘ICRS’, coord_ra, coord_dec),CIRCLE(‘ICRS’, "
+ str(coord.ra.value) + ", " + str(coord.dec.value) + ", "
+ str(radius.value) + " )) = 1 "
“AND refExtendedness = 1 AND detect_isPrimary = 1”
print(query)

df = service.search(query, maxrec=max_rec).to_table().to_pandas()
len(df.coord_ra)

which yields:
E19 Traceback (most recent call last)
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:242, in DALQuery.execute_votable(self, post)
241 try:
→ 242 return votableparse(self.execute_stream(post=post).read)
243 except Exception as e:

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/utils/decorators.py:546, in deprecated_renamed_argument..decorator..wrapper(*args, **kwargs)
544 warnings.warn(msg, warning_type, stacklevel=2)
→ 546 return function(*args, **kwargs)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/table.py:160, in parse(source, columns, invalid, verify, chunk_size, table_number, table_id, filename, unit_format, datatype_mapping, _debug_python_based_parser)
156 with iterparser.get_xml_iterator(
157 source,
158 _debug_python_based_parser=_debug_python_based_parser) as iterator:
159 return tree.VOTableFile(
→ 160 config=config, pos=(1, 1)).parse(iterator, config)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/tree.py:3601, in VOTableFile.parse(self, iterator, config)
3600 else:
→ 3601 vo_raise(E19, (), config, pos)
3602 config.update(self._get_version_checks())

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/exceptions.py:112, in vo_raise(exception_class, args, config, pos)
111 config = {}
→ 112 raise exception_class(args, config, pos)

E19: None:1:0: E19: File does not appear to be a VOTABLE

During handling of the above exception, another exception occurred:

DALServiceError Traceback (most recent call last)
Input In [5], in <cell line: 3>()
1 # For more detailed analysis of results, converting
2 # to a pandas dataframe is often very useful (“df” stands for “dataframe”)
----> 3 df = service.search(query, maxrec=max_rec).to_table().to_pandas()
5 # Use an assertion to make sure we got the correct number of results.
6 # assert len(df) == 15670
7 len(df.coord_ra)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/tap.py:248, in TAPService.run_sync(self, query, language, maxrec, uploads, **keywords)
219 def run_sync(
220 self, query, language=“ADQL”, maxrec=None, uploads=None,
221 **keywords):
222 “”"
223 runs sync query and returns its result
224
(…)
244 TAPResults
245 “”"
246 return self.create_query(
247 query, language=language, maxrec=maxrec, uploads=uploads,
→ 248 **keywords).execute()

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/tap.py:942, in TAPQuery.execute(self)
928 def execute(self):
929 “”"
930 submit the query and return the results as a TAPResults instance
931
(…)
940 for errors parsing the VOTable response
941 “”"
→ 942 return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:244, in DALQuery.execute_votable(self, post)
242 return votableparse(self.execute_stream(post=post).read)
243 except Exception as e:
→ 244 self.raise_if_error()
245 raise DALFormatError(e, self.queryurl)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:253, in DALQuery.raise_if_error(self)
251 if self._ex:
252 e = self._ex
→ 253 raise DALServiceError.from_except(e, self.queryurl)

DALServiceError: 401 Client Error: Unauthorized for url: https://data.lsst.cloud/api/tap/sync

MelissaGraham · September 19, 2022, 10:29pm

Hi Bob,

It’s not clear to me which query statement of the two that you have copy-pasted above you are trying to execute, but they both seem to be working fine for me. I think there is no current error with the TAP service, but rather something in your code.

For example:

from lsst.rsp import get_tap_service, retrieve_query
service = get_tap_service()

query = "SELECT objectId, coord_ra, coord_dec, u_cModelFlux, g_cModelFlux, " + \
        "r_cModelFlux, i_cModelFlux, z_cModelFlux, y_cModelFlux, " + \
        "tract, patch, detect_isPrimary, refExtendedness " + \
        "FROM dp02_dc2_catalogs.Object " + \
        "WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), CIRCLE('ICRS', 62, -37, 0.1)) = 1 " + \
        "AND refExtendedness = 1 AND detect_isPrimary = 1"

results = service.search(query, maxrec=1000)

The above search executes without error, and results are returned.

As another example, the second version of your query statement:

query2 = "SELECT objectId, coord_ra, coord_dec, g_hsmShapeRegauss_e1, g_hsmShapeRegauss_e2, " + \
         "u_ixx, r_ixx, i_ixx, z_ixx, y_ixx, refBand, " + \
         "g_hsmShapeRegauss_flag, g_ixx, g_ixxDebiasedPSF, g_ixxPSF, g_ixxRound, g_ixy, " + \
         "g_ixyDebiasedPSF, g_ixyPSF, g_ixyRound, g_iyy, g_iyyDebiasedPSF, " + \
         "g_iyyPSF, g_iyyRound, shape_flag, shape_xx, shape_xy, shape_yy, " + \
         "tract, patch, detect_isPrimary, refExtendedness " + \
         "FROM dp02_dc2_catalogs.Object " + \
         "WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), CIRCLE('ICRS', 62, -37, 0.1)) = 1 " + \
         "AND refExtendedness = 1 AND detect_isPrimary = 1"

results2 = service.search(query2, maxrec=1000)

The above search executes without error, and results are returned.

I thought maybe the error was stemming from the single line in which you send the TAP search results directly into a pandas table, but for me this line also executes without error, and the dataframe df is filled:

import pandas
df = service.search(query, maxrec=1000).to_table().to_pandas()
len(df.coord_ra)

From the above lines of code, the number 1000 is successfully returned as the length of df.coord_ra.

From here I can recommend to take another close look at your code and try again? And if you’re still getting errors, please do post again in this thread. If and when you do, it would be helpful if you could use the “</>” button to format the pasted material as code (like I’ve done, above). This will help us to see exactly your inputs, reproduce your code, and maybe it will be clearer what the underlying issue is. Sometimes screenshots can be more helpful for us to diagnose the problem, too.

DouglasLTucker · September 19, 2022, 10:55pm

Dear Bob,
I have also been looking more closely at your code. I basically appended it to the end of Tutorial NB 02 and ran it from there. I found 2 issues, but I am still not able to run your queries successfully. I also tried Melissa’s versions of the queries, and I find that they work for me. Following Melissa’s suggestions, it could be an issue with the formatting of the query string that you send to “service.search()”.

BTW, here are the issues I did find:

I got an error trying to run:

coord = SkyCoord(ra=62.0u.degree, dec=-37.0u.degree, frame=‘icrs’)

I found this to work instead:

coord = SkyCoord(62.0, -37.0, frame='icrs', unit='deg')

For continuing lines when defining your query string, I think you need a “+” in addition to a “/”. E.g.,

query = "SELECT objectId, coord_ra, coord_dec, g_hsmShapeRegauss_e1, g_hsmShapeRegauss_e2, " + \
        "u_ixx, r_ixx, i_ixx, z_ixx, y_ixx, refBand, " + \
        "g_hsmShapeRegauss_flag, g_ixx, g_ixxDebiasedPSF, g_ixxPSF, g_ixxRound, g_ixy, " + \
        "g_ixyDebiasedPSF, g_ixyPSF, g_ixyRound, g_iyy, g_iyyDebiasedPSF, " + \
        "g_iyyPSF, g_iyyRound, shape_flag, shape_xx, shape_xy, shape_yy, " + \
        "tract, patch, detect_isPrimary, refExtendedness " + \
        "FROM dp02_dc2_catalogs.Object " + \
        "WHERE CONTAINS(POINT(‘ICRS’, coord_ra, coord_dec),CIRCLE(‘ICRS’, " + \
        str(coord.ra.value) + ", " + str(coord.dec.value) + ", " + \
        str(radius.value) + " )) = 1 " + \
        "AND refExtendedness = 1 AND detect_isPrimary = 1"

instead of

query = "SELECT objectId, coord_ra, coord_dec, g_hsmShapeRegauss_e1, g_hsmShapeRegauss_e2, "
                "u_ixx, r_ixx, i_ixx, z_ixx, y_ixx, refBand, "
                "g_hsmShapeRegauss_flag, g_ixx, g_ixxDebiasedPSF, g_ixxPSF, g_ixxRound, g_ixy, "
                "g_ixyDebiasedPSF, g_ixyPSF, g_ixyRound, g_iyy, g_iyyDebiasedPSF, "
                "g_iyyPSF, g_iyyRound, shape_flag, shape_xx, shape_xy, shape_yy, "
                "tract, patch, detect_isPrimary, refExtendedness "
                "FROM dp02_dc2_catalogs.Object "
                "WHERE CONTAINS(POINT(‘ICRS’, coord_ra, coord_dec),CIRCLE(‘ICRS’, "
                + str(coord.ra.value) + ", " + str(coord.dec.value) + ", "
                + str(radius.value) + " )) = 1 "
                “AND refExtendedness = 1 AND detect_isPrimary = 1”

When I make the above 2 fixes, I still get an error of the sort

DALQueryError: ADQL syntax error: Lexical error at line 1, column 433.  Encountered: "\u00e2" (226), after : ""

which leads me to suspect some weird invisible character in the cut-and-paste query might be causing the problem.

I hope this helps!

So, my basic suggestion: grab Melissa’s versions of the queries!

Best regards,
Douglas

ktl · September 20, 2022, 7:11am

Pasting code directly into the edit box without formatting it as code leads to several problems. Asterisks (as in ra=62.0*u.degree, dec=-37.0*u.degree) often get converted to emphasis (italics). Single quotes (as in 'icrs') may get converted to “smart quotes”, which are illegal in ADQL. I believe this is where the \u00e2 lexical error comes from (it’s the first byte of the UTF-8 representation of the “left single quotation mark”).

But it is legal Python to append strings using concatenation without an operator, even when continued over a line break:

>>> s = "foo" "bar"
>>> print(s)
foobar
>>> s = "foo" \
... "bar"
>>> print(s)
foobar

babel · September 20, 2022, 5:37pm

`I’ve simplified the TAP process in the code below and I still get the same error. It feels like a permissions issue. I can successfully run
assert service.baseurl == “https://data.lsst.cloud/api/tap”
but the error at the bottom says
01 Client Error: Unauthorized for url: https://data.lsst.cloud/api/tap/sync

Also, thank you for the editing suggestions. Is there a Community tutorial I can look at for etiquette? This is a great support network and I want to use it properly.

from astropy import units as u
from astropy.coordinates import SkyCoord
from lsst.rsp import get_tap_service, retrieve_query
service = get_tap_service()

coord = SkyCoord(ra=62.0*u.degree, dec=-37.0*u.degree, frame='icrs')
radius = 0.1 * u.deg

query = "SELECT objectId, coord_ra, coord_dec  " \
        "FROM dp02_dc2_catalogs.Object " \
        "WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec),CIRCLE('ICRS', " \
        + str(coord.ra.value) + ", " + str(coord.dec.value) + ", " \
        + str(radius.value) + " )) = 1 " \
        "AND refExtendedness = 1 AND detect_isPrimary = 1"

max_rec = 1000
df = service.search(query, maxrec=max_rec).to_table().to_pandas()
```
```
WHICH YIELDS:`
```
E19                                       Traceback (most recent call last)
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:242, in DALQuery.execute_votable(self, post)
    241 try:
--> 242     return votableparse(self.execute_stream(post=post).read)
    243 except Exception as e:

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/utils/decorators.py:546, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    544         warnings.warn(msg, warning_type, stacklevel=2)
--> 546 return function(*args, **kwargs)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/table.py:160, in parse(source, columns, invalid, verify, chunk_size, table_number, table_id, filename, unit_format, datatype_mapping, _debug_python_based_parser)
    156 with iterparser.get_xml_iterator(
    157         source,
    158         _debug_python_based_parser=_debug_python_based_parser) as iterator:
    159     return tree.VOTableFile(
--> 160         config=config, pos=(1, 1)).parse(iterator, config)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/tree.py:3601, in VOTableFile.parse(self, iterator, config)
   3600         else:
-> 3601             vo_raise(E19, (), config, pos)
   3602 config.update(self._get_version_checks())

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/astropy/io/votable/exceptions.py:112, in vo_raise(exception_class, args, config, pos)
    111     config = {}
--> 112 raise exception_class(args, config, pos)

E19: None:1:0: E19: File does not appear to be a VOTABLE

During handling of the above exception, another exception occurred:

DALServiceError                           Traceback (most recent call last)
Input In [15], in <cell line: 2>()
      1 max_rec = 1000
----> 2 df = service.search(query, maxrec=max_rec).to_table().to_pandas()

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/tap.py:248, in TAPService.run_sync(self, query, language, maxrec, uploads, **keywords)
    219 def run_sync(
    220         self, query, language="ADQL", maxrec=None, uploads=None,
    221         **keywords):
    222     """
    223     runs sync query and returns its result
    224 
   (...)
    244     TAPResults
    245     """
    246     return self.create_query(
    247         query, language=language, maxrec=maxrec, uploads=uploads,
--> 248         **keywords).execute()

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/tap.py:942, in TAPQuery.execute(self)
    928 def execute(self):
    929     """
    930     submit the query and return the results as a TAPResults instance
    931 
   (...)
    940        for errors parsing the VOTable response
    941     """
--> 942     return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:244, in DALQuery.execute_votable(self, post)
    242     return votableparse(self.execute_stream(post=post).read)
    243 except Exception as e:
--> 244     self.raise_if_error()
    245     raise DALFormatError(e, self.queryurl)

File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.0.0/lib/python3.10/site-packages/pyvo/dal/query.py:253, in DALQuery.raise_if_error(self)
    251 if self._ex:
    252     e = self._ex
--> 253     raise DALServiceError.from_except(e, self.queryurl)

DALServiceError: 401 Client Error: Unauthorized for url: https://data.lsst.cloud/api/tap/sync

ktl · September 20, 2022, 6:14pm

I agree that this appears to be a permission issue. The difference between the base URL and the URL in the error is immaterial; sync is appended to indicate that the query should be executed synchronously.

One possibility might be that your API token has expired. Have you tried logging out and logging back in again?

DouglasLTucker · September 20, 2022, 7:55pm

Thanks, K.-T.!

babel · September 20, 2022, 7:59pm

That was it! Thank you K-T, and also Douglas and Melissa for your help. You’re all great teachers : )