gbdesAstrometricFit: Cholesky decomposition failed

I’m processing a field of several overlapping HSC pointings all in a single tract. step2c fails for the g-band:

lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'gbdesAstrometricFit' on quantum {instrument: 'HSC', skymap: 'skymap_v1', physical_filter: 'HSC-G', tract: 0, band: 'g'} failed. Exception RuntimeError: Cholesky decomposition failed, likely because data is not sufficient to constrain the model

Multiple other bands went through just fine. How should I approach debugging this? Presumably there are some bad exposures that, if I discard, will allow the step to succeed; how can I identify those?

Hi :wave:

My understanding is that the ...: Cholesky decomposition failed, ... is in general due to the fact there is not enough data to constraint the model. Here is some things to look at:

  • The first thing I would look at is the number of visit in this tract and band, and if you have just 1, 2, or 3 visits it might explain the problem, then I think there is not much to do by using the current strategy of fitting astrometry per tract per filter beyond removing the visit in this tract that correspond to the g filter. There is another option I would put at the end of this thread.

  • If there is enough visit, I would check the number of source that is going into the fit. I found in ComCam that some time it was failing because of the fact that some part of the FoV had no input sources, so you had no data to constraint the model. On this specific situation then there is a work to do on understanding why there is no source detected in those specific visit and a work to do on source detection.

If at the end it looks like this is the number of visit the problem, and that GBDES worked on that tract with the other filter, a solution could be to use gbdesAstrometricMultibandFit where it will fit the astrometric solution on a given tract using all filters. See DM-42519 for more information.

Please let me know if you have any questions on this and let me know if I can help on this.

Thank you, @leget for your response. @price, do you still need further assistance with this? Please don’t hesitate to let us know!

Using gbdesAstrometricMultibandFit required a code edit (its ConfigClass is wrong) and addition of the task in the pipeline definition. It’s running forever and ever, possibly due to overly-strict iteration criteria in gbdes (nclip == 0). I’ve returned to running gbdesAstrometricFit on just the g-band after dropping CCDs with bad astrometric solutions. That is also running forever, so I’m playing with modifications to gbdes.

You mean there is a bug that needs to be fixed and needs a Jira ticket?

I believe @leget is working on similar things, and will incorporate the fix I sent on Slack.