Skip to content

Exact match earns its registration: the quieter metrics lose more signal than noise

The sensitivity analysis puts the design's resolution at about two exact-match points, which is where the observed effects are, so the obvious lever is a less noisy outcome. Exact match is binary and maximally variable; the run already records a normalised exact match and an edit similarity. Read over the three seeds under the crossed interval, on the same clusters:

metric OpenStack: effect, half-width, ratio Qt: effect, half-width, ratio
exact match (registered) +0.0171, 0.0232, 0.74 +0.0312, 0.0243, 1.28
normalised exact match +0.0183, 0.0247, 0.74 +0.0298, 0.0255, 1.17
edit similarity -0.0003, 0.0117, 0.03 +0.0059, 0.0092, 0.64

Edit similarity halves the interval and costs five-sixths of the effect. An adapter trained on the right organization is not generally closer in string distance to the reference; it is more often exactly right. The organizational gain lives in the last token as much as the first, which is what an exact-match criterion is for and what a graded string distance averages away.

So the registered metric is the one the data supports, not merely the one chosen first, and the resolution limit is a property of the question rather than of the instrument. The change-averaged estimand does not help either: its Qt interval is wider relative to its effect (0.77 against 1.28).