How to control the number of matches when do astrometry?

[Yuanyu (Yuanyu Hong)]

Recently I run your pipeline to calibrate a TAN-SIP distortion of the mock data, and it works pretty good:
for example it’s match radius converges to 0.01 pixel after only 1 iter:

<match radius> = 40.4 +- 14.87 [74 matches]
Dropping into debugger to allow inspection of display. Type 'continue' when done.
(Pdb) continue
<match radius> = 0.01142 +- 0.07599 [74 matches]
Dropping into debugger to allow inspection of display. Type 'continue' when done.

And the resiudals of the image is:

And I want to know how many stars are necessary for one image(see what will happen if the match number become less), In the above output log, it’s match number is [74 matches]
And I try to achieve the aim by set:

config.astrometry.matcher.numBrightStars = 20

which will infect the max_n_patters in pessimistic_pattern_matcher_b_3D.py(After read the code I think it’s the config I should to modify but it didn’t work.)

So I try to trim the source catalog by modify the code

goodSourceCat = sourceCat 
# to
goodSourceCat = sourceCat[:30] 

in matchPessimisticB.py , it actually reduce the number of match, but here the sourcecatalog is not sorted, it will affect the steps after that.

so I modify the

sorted_source_array = source_array[source_array[:, -1].argsort(), :3] 
# to 
sorted_source_array = source_array[source_array[:, -1].argsort(), :3][:30] 

in pessimistic_pattern_matcher_b_3D.py, but it didn’t work.
So, where should I modify?
Another question:
in the config file of HSC

# Better astrometry matching
config.astrometry.matcher.numBrightStars = 150

with a comment # Better astrometry matching, why the value 150 is better than others? how it’s been confirmed?
Thank you!

[ameisner (Aaron Meisner)]

Elsewhere on this forum there are some suggestions to use the calibrate task’s maxRefObjects parameter to control the resource usage (peak memory, runtime) of the astrometry solving. For this purpose, I try to always use:

-c config.astrometry.matcher.maxRefObjects=3000

I believe the default value is much larger than 3000.

[ameisner (Aaron Meisner)]

Some further discussion of maxRefObjects and other astrometry-related parameters that might be worth exploring is at:

Bus error when running pipelines

FYI, I’ve found a solution for the large memory usage in the calibrate task when processing very dense fields. The high memory usage comes from using very many (50k+) reference catalog sources in the reference matching portion of WCS fitting in AstrometryTask. There is a configurable present in the pipelines that solves this by limiting the number of reference sources used in the reference match: tasks: calibrate: class: lsst.pipe.tasks.calibrate.CalibrateTask config: astromet…

[Yuanyu (Yuanyu Hong)]

Do you have code that make matches unifrom distributed across the CCD?
Recently, I tried to trim the number of matches in astrometry.py, by add matches = matches[:1000], and the matches will feed to wcsFitter

matchRes = self.matcher.matchObjectsToSources(
            refCat=refCat,
            sourceCat=goodSourceCat,
            wcs=wcs,
            sourceFluxField=sourceFluxField,
            refFluxField=refFluxField,
            match_tolerance=match_tolerance,
        )

matches = matches[:1000]

I thought that since you match the source and refrence by the order of magnititude, so I can trim the number of matches in above way.
Another way is limited the matches by limited the reference, and since the source and refrence are matched by the order of magnititude, trim the reference will get the same results as trim the matches(I guess)

config.astrometry.matcher.maxRefObjects = 

but the code it works not like what I thought, since the fitted wcs is not the same for the almost same number of matches by two methods.
for example, if I trim the matches by trim the match array, the result is below

matches = matches[:300]

red line represents 0.01 pixel.
the distortion is large at the top of the image,
but if I trim the matches by set a limit to the number of reference star, the fitted wcs is much more better, and I guess, there may exists some code that used to make matches distributed unifromly but I didn’t find, is that so?
Thank you!
for example, if I limited the matches by limited the reference, and get 260 matches

config.astrometry.matcher.maxRefObjects=500

[ameisner (Aaron Meisner)]

Hi Yuanyu,

Thanks a lot for the updates.

@Yuanyu

Recently, I tried to trim the number of matches in astrometry.py, by add matches = matches[:1000], and the matches will feed to wcsFitter

I don’t think that we’d recommend hacking the pipeline code.

According to this forum post, the sorting by magnitude happens in a file called matchPessimisticB.py (though at this level of detail I suppose the source could have changed over the last year plus).

It seems to me based on your final quiver plot that the maxRefObjects config parameter approach is working as intended?

As part of the single-frame processing, there can be outputs called srcMatch generated that summarize the reference catalog matches found. Those may be interesting to have a look at, as far as assessing spatial uniformity of your reference selections.

I’m not immediately concerned that you ended up with only 260 matches when specifying maxRefObjects=500 — I can check this in more detail later, but I think that the capping of number of reference stars may happen before the cross-match (not after).

Another option you could consider is somehow modifying/trimming your input reference catalogs to only have e.g., a certain range of magnitudes or spatial density.

Lastly, the same forum topic I’ve mentioned previously has some interesting suggestions about making use of the astrometry.referenceSelector and astrometry.sourceSelector tasks. I have not personally worked with those, but they do seem potentially relevant here.