Problem with joining directors in Qserv

Data Management
#1

Dear Qserv devs,

We’ve been trying to do joint queries between equally partitioned director tables. A problem was found possibly due to missing chunks.

For example, if I do:

SELECT … FROM DirectorA AS a, DirectorB AS b WHERE scisql_angSep(a.ra, a.decl, b.ra, b.decl) < 0.0167,

where DirectorA and DirectorB are equally partitioned director tables (I’ve also unified their partition identifiers in CSS).

The query will likely fail, and the log suggests this is due to missing chunks. For example if DirectorA has chunks 400,444 and DirectorB has only 400, Qserv tries to find 444 for DirectorB anyway.

And similar error is also found when we do joint queries between director and child which belongs to another director table.

Currently to make things work, I’m doing some sort of tricks by adding dummy empty chunks for all directors and modifying the qservw_worker.Chunks tables in the data servers. But this is not an ideal. Do you have plans to fix this in future releases? The last test I’ve done was in qserv_12.1-186.

Many Thanks,
Teng

#2

Hi Teng,

thanks you of reporting this problem!

Yes, this is true - this type of joins will only (in the current implementation of Qserv) work for spatial areas where both (or, in general, many) catalogs have chunks. And this is definitively a problem. It can be solved in two ways - the first one you’ve already explored. The other one would be to improve Qserv query scheduler to recognize this type of joins and avoid scheduling queries for chunks with incomplete representation.

We will investigate the best ways to address this problem.

Regards,
Igor

#3

Hi Teng,

This is a known limitation and issue DM-17774 has been created to work on a fix for this bug.

#4

Thanks for the update.

Mike