Looking at persisted metric values with print_metricvalues


Science pipelines developers and users who work with measurements of metrics may be interested in a tool I added to the lsst.verify package this summer. print_metricvalues can output all of the metricvalues stored in a butler collection, sorted by their dataIds. If given two butler repos or collections, it will print out a summary of the differences between the metric values stored in those repos. This can be useful when analyzing metric changes between two runs, without needing to upload results to the SQuaRE/Chronograph dashboard.

I added this tool on DM-34865 as part of our work on the AP performance characterization sprint. It was made specifically to work with the output of running ap_verify on one of the ap_verify datasets, and I have not used it to look at other metric values in other repositories. If you find it useful, please send me feedback on how it behaves with metrics produced from sources other than ap_verify.

Example usage

Here are some examples of using this tool on a run of ap_verify on the ci_cosmos data. See the command line documentation, and the functions in lsst.verify.extract_metricvalues to use this functionality in a notebook.

These examples are all taken from ap_verify runs on the cosmos_pdr2 CI dataset. I’m very curious to see if this tool is useful for others in contexts outside of ap_verify.

Example printing all “value” metrics

$ print_metricvalues cosmos-main/repo ap_verify-output --data-id-keys detector visit

{instrument: 'HSC'}
ap_association.totalUnassociatedDiaObjects: 421.0 ct

detector: 50, visit: 59150
ap_association.numAssociatedSsObjects: 0.0 ct
ap_association.numNewDiaObjects: 268.0 ct
ap_association.numTotalSolarSystemObjects: 0.0 ct
ap_association.numUnassociatedDiaObjects: 0.0 ct
ip_diffim.fracDiaSourcesToSciSources: 0.08153331305141466
ip_diffim.numSciSources: 3287.0 ct
pipe_tasks.numDeblendedSciSources: 504.0

detector: 51, visit: 59160
ap_association.fracUpdatedDiaObjects: 0.09183673469387756
ap_association.numAssociatedSsObjects: 1.0 ct
ap_association.numNewDiaObjects: 171.0 ct
ap_association.numTotalSolarSystemObjects: 1.0 ct
ap_association.numUnassociatedDiaObjects: 178.0 ct
ip_diffim.fracDiaSourcesToSciSources: 0.05608028335301062
ip_diffim.numSciSources: 3388.0 ct
pipe_tasks.numDeblendedSciSources: 534.0

Example printing difference between two runs

$ print_metricvalues cosmos-main/repo ap_verify-output cosmos-DM-34539-fakes/repo
Showing difference of ci_cosmos-DM-34539-fakes/repo#ap_verify-output - ci_cosmos-main/repo#ap_verify-output

{instrument: 'HSC'}
ap_association.totalUnassociatedDiaObjects: 26.0 ct / 372.0 ct

{instrument: 'HSC', detector: 50, visit: 59150, ...}
ap_association.numNewDiaObjects: 21.0 ct / 242.0 ct

{instrument: 'HSC', detector: 51, visit: 59160, ...}
ap_association.fracUpdatedDiaObjects: 0.06349726775956285 / 0.07650273224043716
ap_association.numNewDiaObjects: 19.0 ct / 144.0 ct
ap_association.numUnassociatedDiaObjects: 3.0 ct / 169.0 ct
Number of metrics that are the same in both runs: 11 / 30

Example printing timing metrics

$ print_metricvalues cosmos-main/repo ap_verify-output --kind timing

{instrument: 'HSC', detector: 50, visit: 59150, ...}
ap_association.AssociationTime: 0.001145 s
ap_association.DiaForcedSourceTime: 0.3135 s
ap_association.DiaPipelineTime: 4.738 s
ap_association.LoadDiaCatalogsTime: 0.02981 s
ap_association.MapDiaSourceTime: 0.06324 s
ap_association.PackageAlertsTime: 2.544 s
ip_diffim.DipoleFitTime: 4.052 s
ip_diffim.ImagePsfMatchTime: 27.33 s
ip_isr.IsrTime: 4.782 s
meas_algorithms.SourceDetectionTime: 0.9188 s
pipe_tasks.CalibrateTime: 32.04 s
pipe_tasks.CharacterizeImageTime: 30.54 s
pipe_tasks.ImageDifferenceTime: 36.63 s

{instrument: 'HSC', detector: 51, visit: 59160, ...}
ap_association.AssociationTime: 0.004256 s
ap_association.DiaForcedSourceTime: 0.4046 s
ap_association.DiaPipelineTime: 3.514 s
ap_association.LoadDiaCatalogsTime: 0.05953 s
ap_association.MapDiaSourceTime: 0.06118 s
ap_association.PackageAlertsTime: 1.778 s
ip_diffim.DipoleFitTime: 4.267 s
ip_diffim.ImagePsfMatchTime: 27.52 s
ip_isr.IsrTime: 4.731 s
meas_algorithms.SourceDetectionTime: 0.9358 s
pipe_tasks.CalibrateTime: 36.68 s
pipe_tasks.CharacterizeImageTime: 31.88 s
pipe_tasks.ImageDifferenceTime: 36.88 s

This is fantastic, John! I think it will be super useful for many other contexts. I’ve just started playing around with it, but one initial comment: I was initially confused by the format of outputs when comparing two repos (in fact, I’m still not positive what it’s printing). Might be good to print a message to the screen explaining that the two values are (1) the difference (and whether its repo2-repo1, or repo1-repo2), and (2) the value in one of the repos (which one?).