PILOT RESULT: the exact-match pass rule survives contact with the corpus¶
Job 143496. Qwen2.5-Coder-7B, 201 well-posed OpenStack examples, split by change: 91 changes to 72 train / 19 eval, 156 / 45 examples, zero leakage verified.
| base | adapted | |
|---|---|---|
| exact match | 0.000 | 0.200 |
| normalized exact match | 0.022 | 0.222 |
| edit similarity | 0.184 | 0.822 |
| median prediction length | 344 chars | 52 (reference 63) |
Training: loss 1.58 to 0.40 over 18 optimiser steps, no skipped steps.
What this settles. The open question since the first proxy measurement was whether exact match can discriminate at all on this corpus, given a base model floors at zero. It can. Adaptation lifts it from 0.000 to 0.200 on data the adapter has not seen, which is in the same band as published fine-tuned figures for this task (CodeReviewer 30.32%, CodeT5 24.41%, T5 15.08%) and unsurprising on the low side given 156 training examples against their corpus. The pre-registered pass rule does not need rethinking before 2026-11-20.
Withdrawn: "adaptation fixes the verbosity". This section originally credited the adapter with cutting answers from 344 to 52 characters and called it evidence of learning conventions of form. It was the prompt format. The pilot scored both arms on the raw prompt, which is out of distribution for an instruction-tuned model; the adapter had been trained on that raw format and the base model had not. Job 143892, base model only, same 45 examples:
| base model | exact match | normalized EM | edit similarity | median answer |
|---|---|---|---|---|
| raw prompt | 0.000 | 0.022 | 0.193 | 344 chars |
| chat template | 0.022 | 0.200 | 0.754 | 64 chars (reference 63) |
Two consequences. The base arm's edit similarity and normalized EM were deflated by the format, so every ADAPTED - BASE figure above except exact match is confounded. And under its own template the base model already solves 20% of edits up to whitespace while scoring 2% exact, so the adapter's exact-match gain is largely whitespace and indentation fidelity. That cancels in the RQ1 contrast, where both arms are adapters, and inflates the positive control, where one is not.
A leaked result was nearly reported. The first run with a naive example-level shuffle
gave adapted EM 0.512. 63% of its eval examples shared a change with training data and
20% shared an exact before text. sphragis.corpus.split groups by change precisely to stop
this, and the experiment script bypassed the guard rather than reusing it. Grouping halves
the number, and 0.200 is the one to quote.
Caveats, stated because this number will be quoted. 45 eval examples over 19 changes. Superseded 2026-09-15: the paired cluster bootstrap on this run is +0.200 [+0.086, +0.419], but the run itself is not the one to quote: it skipped dedup, scored a format-confounded base arm, and trained under a loop that never reached one example. The rerun under corrected conditions replaces it. One month, one organization, one seed. And this is not the RQ1 comparison, which contrasts an adapter trained on one organization against one trained on another, evaluated on the first. This measures only that the metric has room to move.