Label audit v2: two blind model raters agree at kappa 0.56, and validity does not differ between the units a contrast compares¶

The first audit's context-dependent class folded two questions together: whether the request can be understood from what the example shows, and whether the rewrite uses names the example does not show. A clear request answered with a project's own constant is a valid label that no prompt alone reproduces, which is house-style knowledge an adapter can learn, not label noise. Rubric version 2 asks the two separately (scripts/label_audit/rubric.md).

A fresh sample of 384 refined v2 examples, 48 per organization, window and placebo half, one per change (label_audit_sample.py, seed 20260923), was blinded under neutral ids in a seeded order with the organization, project, window and half removed (label_audit_blind.py). Two model raters labelled every item alone, blind to each other: rater A on Claude Opus 5.5, rater B on Claude Sonnet 5 (scripts/label_audit/rater-prompt.md). The rubric's worked example is item-001, which is excluded from every figure. A first run of rater B split its batches across copies of itself and was lost before it was committed; it is not reported, and the run below was one rater under an added rule against delegation.

Over 383 items (label-audit-v2.json): the label agrees on 86.4% of items, Cohen's kappa 0.557 [0.454, 0.652]; the outside-names question on 94.8%, kappa 0.825 [0.745, 0.894]. The main disagreement is one boundary: 19 items rater A read as partial and rater B as valid, against 5 the other way. For comparison, two human raters reached 0.758 on 383 comment-generation pairs (arXiv 2607.25851) and 0.56 independently on a nine-label taxonomy (arXiv 2604.23667).

Validity does not differ between the units a contrast compares. Share valid, 95% Wilson interval:

Every interval overlaps every other, so label noise is no route by which an organization or a half would read as having a style it does not have. A human's blind check of rater A is in progress; the human-to-model kappa is what decides how far these model labels can be relied on.