I’ll start out with the general problem that I’m trying to solve, in case there is a better solution that I can’t find, then follow-up with the issues faced with my current implementation.
In order to exactly match catalogs both
meas_extensions_scarlet have a
deblend_peakId key, so that the combination of
(parent, deblend_peakId) uniquely identifies a source. So any time catalogs are built downstream from the same
mergeDet catalog (which means deblending and all of the measurement tasks) we can compare the results of the downstream processing by matching the catalogs on those two keys.
For numpy, pandas, and astropy tables, there is built in functionality to merge catalogs based on a set of matching keys, but as far as I can tell afw.table does not have that functionality. So in my local scripts I just convert the catalogs to pandas DataFrame objects and merge them. However, I need to have a solution that works in pipe_analysis in order to use in the
compareCoaddAnalysisTask, which means I need to do one of the following:
- Merge afw catalogs directly
- Have a fast and efficient way to convert an astropy Table or pandas DataFrame into an afw SourceCatalog (the inverse of the
I’m fine with either solution but neither seems to work right now. I have written a function that takes two input catalogs and uses
numpy.lib.recfunctions.join in order to get the indices needed to match the two catalogs. However, it does not appear that an afw Table can be indexed by a non-contiguous set of indices (e.g.
catalog[np.array([0, 2, 4, 8, 6])]).
So unless there is some hidden API that I don’t know about/can’t find, it would appear that I have to do the matching using a 3rd party data structure and convert it back to an afw.table. But… even that is non-trivial, so I’m wondering if there is some hidden corner of the code that will automatically convert an astropy table into an afw catalog.
Any suggestions would be appreciated. If I end up just having to write a function to convert an astropy Table into an afw SourceCatalog I’ll included it in a future post.