Qserv - using neighbour tables to perform joins?

Data Management
#1

Hi

We’ve been looking at running test queries on a Qserv installation of our UKIDSS data. In our data model we use pre-calculated neighbour tables to join our observations internally and with external data sets. eg we have a source, detection and neighbour table.

For our Qserv test we have each table in a separate director database and the source and detection tables are split up into identical chunks.

If I try and do a query like

select blah from dbs.source as s, dbd.detection as d, dbm.neighbour as n
where s.sourceID=n.sourceID and d.detectionID=n.detectionID

I get

ERROR 4110 (Proxy): Query processing error: QI=?: Failed to instantiate
query: AnalysisError:Query involves partitioned table joins that Qserv does
not know how to evaluate using only partition-local data

the query can be run by adding for example

scisql_angSep(s.ra, s.decl, d.ra, d.decl) < 0.016

but this I believe is then effectively not making use of the pre-existing lookups in the neighbour table and doing lots of trig.

Does anyone know of a work around so that the query uses the neighbour table for efficient joining.

Thanks
Mike

#2

Hi Mike,

There is an element of the Qserv architecture explicitly intended to address this need (efficiently joining between directors via explicit match tables). However, to date the feature has not been often used, and I’m unsure of its current status.

SLAC is closed now for winter break, but we’ll try it out when we get back (1/7), and follow up here with more information for you.

Thanks,
–FritzM.

#3

Hi Mike,

A short update that we’ve created DM-17772 to work on this issue.