Hello,
I’m interested in looking at trailed Solar System objects in DP1 and I’m wondering whether there are a set of recommended flags in the DiaSource table that should be used for filtering out bad observations. Measurements with “trail_flag_edge=True” makes sense to exclude but with the all the different flags to choose from I’m unclear whether I should be ignoring measurements where any of the flags are set to True or if we can be more selective in the filtering process.
Thanks,
Sean
Hi @astro-sobrien, thank you for your question. I’m the Forum watcher this week, but this topic is outside my area of expertise. We’ve opened an internal work ticket (SP-2569) to follow up on this issue and we’ll report back within two weeks. In the meantime, all are welcome to chime in with ideas here.
Hi @astro-sobrien, you’re right that there are a lot of flags in the DiaSource table. We are working on putting together additional guidance on flags, which will be coming in the future.
In the meantime, I took a closer look at the flags in the DiaSource table and how they affect the trailed Solar System objects included in DP1. There are 5,765 DiaSources for the trailed DP1 Solar System objects (which I identified by setting SSObjectId > 0 and trailFlux > 0 in the DiaSource table in the Portal aspect of the RSP). For trailed Solar System objects in DP1, I agree that one of the logical flags to check is trail_flag_edge, although none of the DP1 DiaSources for trailed Solar System objects have this flag equal to True. It also looks like many of the flag columns in the DiaSource table do not have any True entries in them for the trailed Solar System objects.
I also checked to find the number of DiaSources for the trailed DP1 Solar System objects that would get removed when I filtered out all the flags with True entries, and only 53 DiaSources out of the 5,765 trailed Solar System DiaSources were removed, which is <1% of the trailed DiaSources. So if you wanted to be conservative in filtering out all flags with True entries, you would still be left with >99% of the trailed Solar System object DiaSources in DP1.
If you’d like to be more selective in filtering out trailed DiaSources based on specific flags, you can look at the brief description of each flag column in the DiaSource table within the Portal aspect of the RSP, and filter out DiaSources with True flags based on your needs for your science. We have some guidance on flags in the DP1 documentation and, as I mentioned above, we are currently putting together more detailed guidance on flags that will be coming in the future.
I hope this addresses your question. I’m going to mark this as the solution, but please let us know if you need further clarification or guidance.
@sgreenstreet when is the plan to have more documentation about the flags? I wouldn’t know to look in the Known issues section of the documentation to learn about the flags, so perhaps that information can be placed in a spot that it is easier to find when the other updates happen.
From your response it’s not clear if the flags trigger if the full trailed object is covered by a CR or only the part that is identified as the moving object that contributes to the science flux? Understanding how these flags are calculated for DP1 would be helpful.
Hello, I can offer a few additional details for this thread.
The vision/plan is that it will take at least a few months to investigate, characterize, and understand all the flags to the point where detailed descriptions and recommendations for flag use can be released. It’s not a trivial task. This would not go in the “Known issues” section. The vision/plan is for flag information to be discoverable, as it will be embedded throughout the data products documentation, summarized on a new page, and demonstrated in the tutorials.
For the second question about flags for cosmic rays (CRs), in the DP1 schemas the CR-related flags say whether they’re flagging a CR in the footprint or in the center (i.e., a 3x3 pixel region around the centroid), but I don’t think there’s a flag for if a full trailed object is covered by a CR - only if there’s a CR overlapping the source footprint.