Fixing Astrometry Matching

Data Management
,
#1

I’m wrapping up some work that adds a new SIP distortion fitter to our codebase, and some preliminary tests by @boutigny are not as promising as we’d like. The new code might be better than our existing SIP fitter (it’s certainly cleaner and its algorithm is better motivated), but it doesn’t solve all of the problems in matching that he’s seeing in processing CFHT data.

I think the fundamental problem is in the matchers; the hope of the above exercise was that better outlier-rejection in the fitter could help us to work around problems in the matcher, but I think that may have always been an unreasonable hope encouraged by the ugliness of the outlier-rejection iteration in the old one. And the right solution for fixing the matcher is both simple and structurally difficult (maybe impossible) right now: we need to match over full visits, not CCDs.

And that means that if we could get a matcher that works reliably - either by writing a new driver that does process full visits at once, somehow, or by writing a new CCD-level matcher - we’d be able to really clean up meas_astrom:

  • We could drop the ANetAstrometry and ANetBasicAstrometry tasks, which use astrometry.net to do the non-SIP part of the WCS fitting as well as the matching. Because astrometry.net combines those two operations, these tasks don’t fit into the same interface that other matchers and fitters use, and because they rely on the old SIP-fitting code to do the distortion part, keeping them around means we have to keep the old SIP-fitting code around (or update them to the new SIP-fitting code, which would then involve code duplication since they don’t use the same interfaces as the new fitters). On top of that, the algorithmic logic in these is horribly contorted - we run ANetBasicAstrometry in a context in which it thinks it’s fitting the full WCS but it’s actually fitting a temporary one using pixel positions that have already had the optical distortion removed, resulting in confusing log messages and diagnostic plots. But we can’t remove them yet, because they apparently still succeed in some cases where our default matcher fails.

  • We now have two SIP fitters. I believe the new one is better than the old one, but it’s hard to test that as long as we’re limited by the matchers. If we could verify that, and we could remove or rewrite the SIP fitting part of the astrometry.net tasks, we could remove the old SIP fitter.

  • We’re still using astrometry.net-format catalogs even when we’re not using the astrometry.net solvers. My understanding is that the new reference catalog loaders are now ready to be used, and we’re still using astrometry.net simply because that’s the format most of our reference catalog datasets are currently in. Converting all of those is obviously a different issue than getting a reliable matcher, but it’d be very nice to get that done and get away from astrometry.net (and its installation difficulties and custom LSST Swig interface) entirely. @KSK, is it just a matter of updating those reference catalog packages, or am I unaware of additional work that needs to be done here?

On top of that, someday I hope we’ll just be able to use the jointcal fitter for single-visit astrometry fitting as well. I don’t think its matcher will ever work in that role, however, because it doesn’t try to look for the unknown offset that’s necessary when working with approximate telescoped-provided WCSs.

For now, I don’t know what to do here. This really is one of our tallest poles in terms of having a reliable pipeline (perhap mostly on CFHT, but we’ve had problems with matching on HSC in the past too). But if the solution is going to be full-visit processing, I think we may be blocked by SuperTask (etc) work that’s not really even been started, let alone nearing completion. I don’t think it’s viable to continue to try to solve this problem by processing individual CCDs - it just isn’t worth our time. I honestly think we’d be better off writing an alternate driver that avoids CmdLineTask and uses multiprocessing to process all the CCDs in a visit together, even if we’d ultimately end up throwing it away in the glorious multithreaded GIL-releasing SuperTask future.

I have heard, however, that @ctslater may have thrown together a new matcher based on a Pan-STARRS algorithm that seemed to work better than our existing one. While I don’t want us to put a lot of work into any single-CCD matcher, even one that’s moderately far along, if it is good enough allow us to get off of astrometry.net and adequately test the SIP fitters, that could be a huge win. @ctslater, is that something we could easily make generally available?

#2

@jbosch I haven’t fully parsed your message, but I wanted to say that we have a new postdoc working on this very problem as we speak. @cmorrison is both looking at the “simple shift” matcher and also looking at the sensitivities of the OPM-B matching algorithm. We should know more in a couple of weeks.

#3

Yay, full-visit processing. It definitely seems like it’ll be the most robust solution for things like this.

However, the jointcal fitter is going to want to be full-visit exposure as well, in the future. But yes, jointcal’s matcher is not really designed for searching a large probability space.

As much as I don’t like throwing away work, this may be a viable option.

#4

I’m confused by this. I understand why full focal plane fitting is conceptually superior, but I don’t know what suggests that single CCD fitting is infeasible even with improvements to the matcher. Are we talking about catastrophic matcher failures, or just that the matcher isn’t finding all the potential matches that are present in the data?

Also just to be sure, I suspect @boutigny is matching against Gaia? If not, I would be interested in knowing if that changes the result.

#5

I’m talking about catastrophic failures, in which we either don’t get enough matches to do any kind of fit, or we have so many bad matches we can’t recover from it with any kind of outlier rejection.

I’m not certain that single CCD matching is infeasible in general - it just seems like a much harder problem to solve than full-visit matching, and given the possibility that it’s infeasible on some datasets we care about, I’m worried about playing whack-a-mole.

#6

Don’t over-estimate how easy it’s going to be to do full-visit matching. All matching algorithms require that you know the distortion. If your guess is slightly off (e.g., because working at low altitude) then you’re going to have trouble over the full focal plane simply because the domain is larger which is going to amplify any little imprecisions. In that case, the single CCD matching is going to be easier because of the limited domain, essentially approximating the distortion piecewise. Maybe a multiple-step process (e.g., CCD matching, fit entire visit, match for CCDs that originally failed to match, fit entire visit) would be better than full-visit matching.

1 Like
#7

I’ve had a bit fuller read and here are some comments.

I don’t think we can give up on single CCD matching for AP quite yet. It is very clearly baked into our resourcing and pipelines for AP that we will deal exclusively in CCD level operations. Even talk of doing raft level crosstalk corrections makes people start to get a little bit antsy. So, though it may be the right thing for DRP, I think we have to spend some time on single sensor astrometry for AP.

As I said before, we have a new postdoc looking at the matcher as we speak. He will both look for bugs in our implementation of OPM_B and look at the Pan-STARRS algorithm. I hope we will see significant improvement because of his work.

The fact that astrometry.net succeeds where our matcher fails means we have a bug in our matcher. I think we’ve known this for some time. I firmly believe that if we constrain ourselves to reasonable situations: i.e. < 5 arcmin pointing error, we should do no worse than the astrometry.net matcher.

I think we will know more when we have fixed the single CCD matcher, but I think making the new SIP fitter the default would be a good step.

@ctslater has already produced the new version of the reference catalogs for the GAIA catalog, so has some knowledge about how to do this. I also think @rowen may have looked at doing this for the SDSS catalogs, but I’m not sure how far he got. There is no reason not to use new style reference catalogs that I know of.

I hope I’m not being too optimistic about the reliability of single CCD matching, but it really does have big implications for the L1 system.

#8

I haven’t been able to find the time to test with the new Gaia catalog. I will do it as soon as possible.

#9

While I understand the mid/long term strategy to switch to full focal plane astrometry fit, I am really concerned if we give up with the individual CCD astrometry now. Having a correct astrometry is the minimum if we want to test the science pipelines. @nchotard and I are going to spend some time in testing the matcher in order to try to understand what is going wrong.
To give a sense of the current problem, here is a portion of an image (top part) where the WCS has been computed with the default matcher + the new fitter from Jim and with the SDSS DR9 reference sources superimposed.

793169_33_new_sn10.png1855×829 835 KB

And here is the same image with the astrometry.net matcher + old fitter :
793169_33_anet_default.png1855×829 835 KB

Nevertheless, I want to stress that we should not overestimate the size of problem and in 99% of the cases the astrometry fit is giving a correct answer. This is also the reason why I think that now that we have a healthy fitter (thanks Jim), fixing the matcher should not be too complicated.

Unfortunately I have very little time to work on this at the moment, things will be better after the Pittsburgh hack week.

Finally I would also like to express some concerns with the SuperTask if it means that we have to give up with the sequential CmdLine as not all computing infrastructures are sized for large multi-node tasks. If this is becoming a pre-requisite to run LSST pipelines, it needs to be planned a little bit in advance.

#10

@boutigny As I’ve mentioned before, @cmorrison is starting to look at this at UW now. Can you share with him the the data to reproduce this example? It’s very useful to have data that fail.

#11

So, overall it seems I was a bit premature in proposing that we give up on single-CCD matching, and I didn’t realize there was already work underway to understand and fix our matchers.

I’ve got a few more replies to specific comments I’ll try to cover together:

Operationally, I think that multi-step process still amounts to full-visit matching, though I agree it may be a better algorithm that a naive match to all of the sources in a visit (and I can imagine even smarter algorithms in between those two

But your point that we shouldn’t expect matching to automatically work better if we give it a larger area is a very good one.

I have to admit I wasn’t thinking at all about AP’s CCD-parallel processing requirements at all here, mostly because I actually wasn’t thinking about LSST data at all (which I expect to be easier to solve than the heterogeneous precursor data we’re dealing with now, because we’ll have better characterizations of the optical distortion, a better reference catalog, and hopefully a better initial estimate from the observatory).

I certainly understand that AP would like to avoid the complexity of full visit processing, and that’s a solid argument for putting quite a bit more work into a single-CCD matcher. If it does prove difficult, though, I think it might be worth thinking about where the complexity tradeoff would be in adding full-visit processing to AP. I suspect the added complexity mostly lands in the Task framework or workflow system in terms of supporting inter-node IPC, but DRP might need that anyway. And if our hardware has enough cores/node to process a full visit on a single node, there may actually be no tradeoff at all.

Interesting. I honestly don’t know enough about the matcher algorithms to judge, but I did wonder whether the astrometry.net algorithm might simply be a better algorithm even for small pointing error.

I do think requiring bigger hardware (more memory/node) is likely inevitable for future versions of the pipeline, not just for full-visit processing but multi-band coadd processing as well. We’ll try very hard to maintain the ability to process subsets of the data (e.g. processing a subset of the focal plane as if it was the entire camera) to allow for some development and testing to happen on small machines. This is really being driven by the needs of the algorithms, not SuperTask - which is right now just a vague term for the future middleware that might allow the kind of processing the more sophisticated algorithms need. There’s a chance we’ll be able to preserve the ability to do full-scale processing on small machines with a lot of swapping, but I’m not convinced it’s worth our time trying to support that other than at the OS level.

#12

@ksk @cmorrison

The raw data file is on lsst-dev in /lsst8/boutigny/793169p.fits.fz and the problematic ccd is 33

#13

I think it’d be premature to design the AP to assume that per-CCD astrometric matching always works. Although I can of course see why this is simpler in the short-run, it’s much harder to retrofit something like this later. I can well believe that down in the plane after a long slew when the seeing is especially good/the transparency is especially bad/… that at least some CCDs will fail – and that we should start planning for this failure mode now.

It’s not trivial, but a lot of getting robust astrometric solvers isn’t trivial and we need to face this. For example, when the seeing is very good you need a catalogue that has a fainter bright limit (or you need to work on algorithms for centroiding saturated stars).

On another (but related) topic, I’m excited that UW is starting to look at matchers, and understanding exactly how and why they fail. The tools that you build to do this will be an important part of the quality analysis systems that we are building. @mjuric has clarified that such tooling is the pipeline groups’ responsibility – which means that we have to be careful not to write them as throw-away components.

#14

I very much agree that we need to work on this, I was only pushing back that we can’t throw away single chip matching/solving completely. It will be perfectly fine for much of the data we take and almost all of the precursor data we are using. It’s also a good place to start with getting a handle on the tricky parts.

Thanks for bringing this up. I’ll make sure @cmorrison keeps this in mind as he’s doing his tests.