This is a general question about table manipulation prompted by a specific issue. My broad question is: is the typical LSST stack user expected to stop using astropy/pandas tables altogether or convert between then and afw.table?
I am trying to make a calibration catalogue for VISTA/HSC by taking the PanSTARRS HSC calibration catalogue and merging in 2MASS. The cross matching and stacking etc is trivial with astropy.table but the result will not be loaded by the stack. I am currently hacking it by altering the fits ‘TFORM’ etc using astropy.io.fits. Is this what most people are doing or should I be writing new cross match etc functionality (or does it already exist) using afw.table? The absolute easiest would be some sort of ‘from_astropy’ functionality.
I followed the very old discussion here but can’t find any example code for basic table manipulation elsewhere.
I’ve just created a GitHub repository that includes the code that I used to filter the PanSTARRS reference catalogue to include sources brighter than 19th mag. There’s also a README file in there.
Great thanks very much. I think it is just the afwTable formatting stuff that is breaking processCcd so the table manipulation stuff will be very helpful!
We definitely don’t encourage (let alone require) everyone to convert external tables to afw.table, and we’re actually trying to reduce our own reliance on afw.table.
But our processing pipelines currently always start with afw.table objects and convert to pandas.DataFrame in later steps, so we’ve only had to do that conversion in one direction (and not the direction you need). Our effort to reduce our own afw.table usage mostly consists of trying move that conversion point earlier.
If I understand your case correctly, you’re trying to produce a reference catalog our pipelines can use, and that’s well within the steps where it’s still just afw.table, so we don’t have any great solutions for you. If you have something working that you understand, it may be best to just live with it, but we can try to answer afw.table questions directly if you’d prefer to use those APIs.