Hi folks, as part of the HUN-KON-S1 contribution for performing forced photometry on Solar System moving objects over (potentially years) of past images along the arc, we need to intersect large numbers of positions from an ephemeris with potential Rubin images (using DP0.2 as the proxy currently). We are currently doing this via the RSP TAP to query ivoa.ObsCore and the queries look like this (time/position windows truncated):
SELECT
lsst_visit,
lsst_detector,
lsst_ccdvisitid,
lsst_band,
s_ra,
s_dec,
t_min,
t_max,
s_region
FROM ivoa.ObsCore
WHERE calib_level = 2
AND (lsst_band = ârâ OR lsst_band = âgâ)
AND t_max >= 59802.415775
AND t_min <= 59863.333979
AND (
(t_min >= 59802.415775
AND t_max <= 59802.416122
AND CONTAINS(POINT(âICRSâ, 58.138352, -41.696853), s_region) = 1)
OR
(t_min >= 59803.360786
AND t_max <= 59803.361133
AND CONTAINS(POINT(âICRSâ, 58.458592, -41.695724), s_region) = 1)
OR
(t_min >= 59803.362719
AND t_max <= 59803.363066
AND CONTAINS(POINT(âICRSâ, 58.459217, -41.695733), s_region) = 1)
OR
(t_min >= 59805.314816
AND t_max <= 59805.315163
AND CONTAINS(POINT(âICRSâ, 59.102676, -41.698524), s_region) = 1)
)
Performance is very slow or just crashes in the query with large numbers of points needed to resolve a long arc.
Weâve investigated other constraints supported by ADQL such as INTERSECTS with POLYGON shapes (since moving objects are often traveling along relatively long but thin paths) based on documentation for the Gaia TAP service. However it seems like support for other than POINTs isnât available in RSP:
DALQueryError: Query Error: UnsupportedOperationException:53 is not a supported coordinate system.
Are there are (documentedâŚ?) alternatives using the Butler we should use instead or stick with RSP TAP and find workarounds, better queries or get additional support added (I appreciate âscope creepâ is the last thing wanted right nowâŚ). Being able to investigate past behavior of newly discovered solar system objects or newly detected outbursts from alerts, is going to be critical for a lot of the time-domain/alert-based solar system science the SSSC wants to do with Rubin.
Hi Tim, I noticed you asked about alternatives using the Butler. Since the objective is to do forced photometry, and the butler is optimized for image analysis with the LSST Science Pipelines (like forced photometry), I think a butler-based solution make sense here.
Butler queries do accept statements like detector_region OVERLAPS user_region. It is possible to establish a polygon and a timespan, and query the butler for all visit images (calexps, in DP0.2) in that timespan. Hereâs an example that returns 70 DP0.2 visit images (calexps) based on the example constraints in your post.
from lsst.daf.butler import Butler, Timespan
import lsst.sphgeom as sphgeom
from astropy.time import Time
butler = Butler('dp02', collections='2.2i/runs/DP0.2')
polygon = """POLYGON 58.138352 -41.696853
58.458592 -41.695724
58.459217 -41.695733
59.102676 -41.698524"""
region = sphgeom.Region.from_ivoa_pos(polygon)
time1 = Time(59802.415775, format="mjd", scale="tai")
time2 = Time(59863.333979, format="mjd", scale="tai")
timespan = Timespan(time1, time2)
del time1, time2
dataset_refs = butler.query_datasets("calexp",
where="band.name IN ('g', 'r') AND \
visit.timespan OVERLAPS timespan AND \
visit_detector_region.region OVERLAPS region",
bind={"timespan": timespan,
"region": region})
print(len(dataset_refs))
If this example is the kind of thing you were looking for, could you mark this post as the solution? If it wasnât quite what you needed, we can keep this topic thread open for further discussion.
Hi @talister, can you confirm if @MelissaGrahamâs response provided what you were looking for? If so, could you please mark it as the solution? If you still need more information, please provide further clarification and we can keep this thread open for additional discussion.
Sorry @sgreenstreet@MelissaGraham Iâm slightly fronting this question for our Hungarian colleagues⌠Iâve pinged them about @MelissaGrahamâs reply and asked if it helped but havenât heard back yet - in theory we have a stand-up meeting tomorrow, so possibly more news after thatâŚ
Hi folks, Laszlo (the primary developer on HUN-KON-S1) reports that the new code fragment is working well for him and speeds things up considerably vs TAP, but he has been unable to find any additional details on building and using butler queries beyond the DP02_04 tutorial. Is this something that there could potentially be more documentation for in the future once the First Look and DP1 release excitements calm down ?
Thanks Tim. Iâm going to mark my reply as the solution but as always, open up a new ticket anytime for new or continuing issues.
Yes, with DP1 we have a few more butler tutorials. It would be interesting to hear more about the types of queries needed and what demonstrations would be most useful, too.