Hi
I’m trying a notebook TAP query, joining DiaObject and ForcedSourceOnDiaObject, using service.search
select f.*
from dp02_dc2_catalogs.DiaObject as o join dp02_dc2_catalogs.ForcedSourceOnDiaObject as f on o.diaObjectId=f.diaObjectId
where o.nDiaSources > 389 or (o.nDiaSources = 389 and o.diaObjectId <= 1253460847850684465 )
After some minutes it fails with the below output. Not sure if it’s because I’m hitting too much data being returned or something I’m doing wrong with the query structure itself, a similar query of DiaObject joined with DiaSource works. The query that fails seemed to work through the portal initially but when I increased the row limit from 50,000 to 5,000,000 it seems to be hanging. I’m also running the query as a count(*) through the portal and that’s hanging too.
I suspect user error but if anyone has any ideas what’s causing this that’d be useful.
Thanks
Mike
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/pyvo/dal/query.py:241, in DALQuery.execute_votable(self, post)
240 try:
--> 241 return votableparse(self.execute_stream(post=post).read)
242 except Exception as e:
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/astropy/utils/decorators.py:604, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
602 warnings.warn(msg, warning_type, stacklevel=2)
--> 604 return function(*args, **kwargs)
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/astropy/io/votable/table.py:177, in parse(source, columns, invalid, verify, chunk_size, table_number, table_id, filename, unit_format, datatype_mapping, _debug_python_based_parser)
174 with iterparser.get_xml_iterator(
175 source, _debug_python_based_parser=_debug_python_based_parser
176 ) as iterator:
--> 177 return tree.VOTableFile(config=config, pos=(1, 1)).parse(iterator, config)
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/astropy/io/votable/tree.py:3858, in VOTableFile.parse(self, iterator, config)
3856 config["_current_table_number"] = 0
-> 3858 for start, tag, data, pos in iterator:
3859 if start:
ValueError: 1:0: syntax error
During handling of the above exception, another exception occurred:
DALServiceError Traceback (most recent call last)
Cell In[37], line 23
21 # source_results = service.search(source_query)
22 print("querying Forced",forced_query)
---> 23 forced_results = service.search(forced_query)
24 if loop==1:
25 break
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/pyvo/dal/tap.py:257, in TAPService.run_sync(self, query, language, maxrec, uploads, **keywords)
228 def run_sync(
229 self, query, language="ADQL", maxrec=None, uploads=None,
230 **keywords):
231 """
232 runs sync query and returns its result
233
(...)
253 TAPResults
254 """
255 return self.create_query(
256 query, language=language, maxrec=maxrec, uploads=uploads,
--> 257 **keywords).execute()
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/pyvo/dal/tap.py:1076, in TAPQuery.execute(self)
1062 def execute(self):
1063 """
1064 submit the query and return the results as a TAPResults instance
1065
(...)
1074 for errors parsing the VOTable response
1075 """
-> 1076 return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/pyvo/dal/query.py:243, in DALQuery.execute_votable(self, post)
241 return votableparse(self.execute_stream(post=post).read)
242 except Exception as e:
--> 243 self.raise_if_error()
244 raise DALFormatError(e, self.queryurl)
File /opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/site-packages/pyvo/dal/query.py:252, in DALQuery.raise_if_error(self)
250 if self._ex:
251 e = self._ex
--> 252 raise DALServiceError.from_except(e, self.queryurl)
DALServiceError: for https://data.lsst.cloud/api/tap/sync