Why the DP1 Object table is missing some very obvious objects?

Support
#1

Hi all,

I’m trying to retrieve coordinates for all objects from the dp1.Object table within the footprint of a visit . Here’s a minimal example:

from lsst.rsp import get_tap_service
from lsst.daf.butler import Butler

def ivoa2adql(ivoa_string: str, frame: str = "ICRS") -> str:
    figure, *coords = ivoa_string.strip().split()
    coord_string = ", ".join(f"{c:.7f}" for c in map(float, coords))
    return f"{figure}('{frame}', {coord_string})"

butler = Butler("dp1", collections="LSSTComCam/DP1")
dataset_refs = butler.query_datasets(
    "visit_image",
    where=f"visit.id = 2024120200214 and detector = 4",
)
visit = butler.get(*dataset_refs)

service = get_tap_service("tap")
sources = service.search(
    f"""SELECT coord_ra, coord_dec, 
    FROM dp1.Object
    WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec),
    {ivoa2adql(visit.getConvexPolygon().to_ivoa_pos())}) = 1"""    
).to_table()

Here is a plot of the returned objects overlaid on the visit image:

image
image1342×1245 189 KB

As you can see, there are noticeable gaps where objects are missing even though the image contains obvious stars in those regions.

Is this expected behavior? Could this be due to selection cuts in the Object table, incomplete processing (possibly due to background), or some issue with the polygon query itself?

Appreciate any insights or suggestions!

As always, thank you!

#2

How are you doing the plotting? It’s not entirely clear from your image that the circles line up with any sources.

If you display the image in Firefly and do a catalog query in firefly and overlap the catalog, do they line up?

#3

Hi @deppep, thanks for posting this question.

First of all, when displaying a visit image, it is Source table that contains all sources detected in a visit image. The Object table contains all detections in the deep coadd images, and so it is not expected that all Objects would appear in a visit image.

But that’s not the primary factor causing the effect you see. This visit image is in the Seagull Nebula, and your guess of “incomplete” processing due to the background is essentially the answer. It is not due to an issue with your polygon query, I checked it and I think that works ok.

Object and Source detections for images of the Seagull Nebula are known to be incomplete due to the high background causing issues with the deblender. This tutorial notebook, 301.7. Seagull Nebula, provides an overview and especially Section 5.2 demonstrates the incomplete detections.

I also put together this demo using your particular visit image, showing the detected Sources (orange) and the NOT_DEBLENDED mask plane (blue) to show that the undetected stars are in the regions where the deblender struggled due to high background (code for copy-paste).

Screenshot 2025-10-03 at 6.14.25 PM
Screenshot 2025-10-03 at 6.14.25 PM1485×869 159 KB

As a final note, the deblender’s algorithms and how they’re applied to LSST data are still a work-in-progress, and these effects we see in DP1 aren’t necessarily representative of how the deblender will work in future data releases.

I think that answers your question so I’m going to mark this reply post as the solution, but please don’t hesitate to respond or to open a new topic for new/related issues any time.

Code for easy copy-paste into a notebook.

Import packages.

from lsst.daf.butler import Butler
import lsst.afw.display as afwDisplay
import numpy as np

Instantiate Butler and Firefly.

butler = Butler("dp1", collections="LSSTComCam/DP1")
afwDisplay.setDefaultBackend("firefly")
afw_display = afwDisplay.Display(frame=1)

Retrieve and display visit image.

visit_image = butler.get('visit_image', visit = 2024120200214, detector = 4)
afw_display.mtv(visit_image)

Retrieve and mark sources.

source = butler.get('source', visit = 2024120200214)
tx = np.where(source['detector'] == 4)[0]
with afw_display.Buffering():
    for i in tx:
        afw_display.dot('o', source[i]['x'], source[i]['y'], size=20, ctype='orange')

Set mask plane transparency.

afw_display.setMaskTransparency(100)
afw_display.setMaskTransparency(80, 'NOT_DEBLENDED')
#4

Thank you a lot for your replies @timj and @MelissaGraham!

How are you doing the plotting? It’s not entirely clear from your image that the circles line up with any sources.

I was using this function for plotting. I didn’t write about it because we made sure that the markers were making sense checking only the brightest objects.

First of all, when displaying a visit image, it is Source table that contains all sources detected in a visit image. The Object table contains all detections in the deep coadd images, and so it is not expected that all Objects would appear in a visit image.

I am aware but for this application we were interested in constant, bright (snr > 200) sources, so the choice of querying the Object table. This was for checking correlations between false detections and bright stars.

I also put together this demo using your particular visit image, showing the detected Sources (orange) and the NOT_DEBLENDED mask plane (blue) to show that the undetected stars are in the regions where the deblender struggled due to high background (code for copy-paste).

Thank you so much Melissa. Your demo is very clear and the gaps in the picture makes sense now. Will keep in mind.

Till next time!!

1 Like