Structurally ill-posed examples: 27% of the corpus¶
Independent of the metric question, and actionable now. Inspecting predictions against references showed several examples no model could answer, so the whole corpus was profiled:
| share | |
|---|---|
after is empty, a pure deletion |
21.5% |
before is empty, a pure insertion |
4.4% |
after more than 5x before, the hunk over-captured |
4.0% |
after under 0.2x before |
1.5% |
| plausibly well-posed | 73.1% |
The length ratio runs to a maximum of 1148x: one case anchors on a single line and pairs it against a thirty-line Ansible block.
Pure deletions are the large category and the clearest problem: the target is the empty string, which a generative model cannot produce naturally and which exact match then scores as a miss for a structural reason. Other cases are under-determined rather than hard, for instance a comment asking why a copyright year changed where the correct year (2020) never appears in the prompt.
These are the misaligned pairs arXiv:2607.25851 describes. Filtering them raised edit similarity from 0.555 to 0.663, so the model does produce closer output on well-posed input, though exact match stayed at zero for the base-model reason above.