Exact match on real hunks: a base-model floor, not a broken metric¶

First run of the whole measurement path over real examples rather than toy ones. Qwen2.5-Coder-1.5B, 30 real OpenStack hunks, 12 changes, no adapter:

On three hand-made toy examples the same model scored 0.67 exact match. On real hunks it scores zero.

Why this matters more than a weak proxy result. The Stage 1 protocol makes exact match the single binding metric and lists non-degeneracy as an outcome-neutral test: exact match is neither 0 nor 1 for every condition on both test sets. That test is designed to halt the study, and on this proxy it fails. If the registered 7B also floors at or near zero on real hunks, the gate cannot detect an organization-specific difference no matter how large one is, and RQ1 returns a null caused by a floor effect rather than by an absence of fingerprints.

Edit similarity of 0.555 says the model is producing related output, not noise. The problem is the strictness of the oracle against real multi-line, context-dependent hunks, not the model failing to engage with the task.

Correction, same session. The paragraph above was written before checking what fine-tuned models actually score, and it overstated the risk. research(2026-09): CodeReviewer reports EM 30.32% on code refinement, against T5 at 15.08% and CodeT5 at 24.41%, with ensembles reaching 36.32%. So exact match is demonstrably attainable on this task in the 15-36% band once a model is fine-tuned for it.

That reframes the measurement above entirely. A base model, zero-shot, with no adapter, scoring EM 0 is the expected result rather than an alarm. The study compares adapted models, and adaptation is precisely what lifts EM off the floor. The design is not broken.

What survives is a narrower and still real check: the pilot has to confirm the adapted 7B clears the floor on this corpus, since CodeReviewer's 30% is on its own differently curated benchmark. If it does not, the pass rule needs rethinking before 2026-11-20. The prior from the literature says it should.

What this changes. The pilot's first job is no longer sample-size estimation, it is confirming the adapted 7B clears the exact-match floor on this corpus. That has to be settled before 2026-11-20, because the pass rule is what gets pre-registered and it cannot be loosened afterwards without deviating from the protocol. Options if the 7B also floors, all of which are pre-registration decisions rather than post-hoc rescues: bind the rule to normalized exact match instead; restrict to single-line hunks where exact match is attainable; or keep exact match and pre-register a sample size large enough for a very small rate to be estimable.

Caveats, stated because the number will be quoted: 1.5B is far weaker than the registered 7B, 30 examples is a small sample, and these are base-model scores with no adapter, where the study compares adapted models. This is an early warning, not a result.