I’m wrapping up some work that adds a new SIP distortion fitter to our codebase, and some preliminary tests by @boutigny are not as promising as we’d like. The new code might be better than our existing SIP fitter (it’s certainly cleaner and its algorithm is better motivated), but it doesn’t solve all of the problems in matching that he’s seeing in processing CFHT data.
I think the fundamental problem is in the matchers; the hope of the above exercise was that better outlier-rejection in the fitter could help us to work around problems in the matcher, but I think that may have always been an unreasonable hope encouraged by the ugliness of the outlier-rejection iteration in the old one. And the right solution for fixing the matcher is both simple and structurally difficult (maybe impossible) right now: we need to match over full visits, not CCDs.
And that means that if we could get a matcher that works reliably - either by writing a new driver that does process full visits at once, somehow, or by writing a new CCD-level matcher - we’d be able to really clean up
We could drop the
ANetBasicAstrometrytasks, which use
astrometry.netto do the non-SIP part of the WCS fitting as well as the matching. Because
astrometry.netcombines those two operations, these tasks don’t fit into the same interface that other matchers and fitters use, and because they rely on the old SIP-fitting code to do the distortion part, keeping them around means we have to keep the old SIP-fitting code around (or update them to the new SIP-fitting code, which would then involve code duplication since they don’t use the same interfaces as the new fitters). On top of that, the algorithmic logic in these is horribly contorted - we run
ANetBasicAstrometryin a context in which it thinks it’s fitting the full WCS but it’s actually fitting a temporary one using pixel positions that have already had the optical distortion removed, resulting in confusing log messages and diagnostic plots. But we can’t remove them yet, because they apparently still succeed in some cases where our default matcher fails.
We now have two SIP fitters. I believe the new one is better than the old one, but it’s hard to test that as long as we’re limited by the matchers. If we could verify that, and we could remove or rewrite the SIP fitting part of the
astrometry.nettasks, we could remove the old SIP fitter.
We’re still using
astrometry.net-format catalogs even when we’re not using the
astrometry.netsolvers. My understanding is that the new reference catalog loaders are now ready to be used, and we’re still using
astrometry.netsimply because that’s the format most of our reference catalog datasets are currently in. Converting all of those is obviously a different issue than getting a reliable matcher, but it’d be very nice to get that done and get away from
astrometry.net(and its installation difficulties and custom LSST Swig interface) entirely. @KSK, is it just a matter of updating those reference catalog packages, or am I unaware of additional work that needs to be done here?
On top of that, someday I hope we’ll just be able to use the
jointcal fitter for single-visit astrometry fitting as well. I don’t think its matcher will ever work in that role, however, because it doesn’t try to look for the unknown offset that’s necessary when working with approximate telescoped-provided WCSs.
For now, I don’t know what to do here. This really is one of our tallest poles in terms of having a reliable pipeline (perhap mostly on CFHT, but we’ve had problems with matching on HSC in the past too). But if the solution is going to be full-visit processing, I think we may be blocked by
SuperTask (etc) work that’s not really even been started, let alone nearing completion. I don’t think it’s viable to continue to try to solve this problem by processing individual CCDs - it just isn’t worth our time. I honestly think we’d be better off writing an alternate driver that avoids
CmdLineTask and uses multiprocessing to process all the CCDs in a visit together, even if we’d ultimately end up throwing it away in the glorious multithreaded GIL-releasing
I have heard, however, that @ctslater may have thrown together a new matcher based on a Pan-STARRS algorithm that seemed to work better than our existing one. While I don’t want us to put a lot of work into any single-CCD matcher, even one that’s moderately far along, if it is good enough allow us to get off of
astrometry.net and adequately test the SIP fitters, that could be a huge win. @ctslater, is that something we could easily make generally available?