Skip to content

Planned before any run: an open-weight rater at pinned weights on the label audit

The October 2026 hardening plan (ROADMAP Plan H, on the GitHub-route branch): raters A and B ran through an API, so the audit cannot be rerun on them once they are retired. scripts/label_audit_open_rater.py adds a rater anyone can rerun: Qwen3-14B (Apache 2.0, revision 40c0698), greedy, thinking off, labelling the same 383 audited examples under the committed v2 rubric and shown only the fields A, B and the human saw. Its agreement with each of them (raw agreement, kappa, AC1 and specific agreement, on the five labels, collapsed to valid or not, and on outside_names) is read with the audit's own statistics, each pair on the items both raters answered (the human's 152 checked items for the human pair). Reading, fixed now: it is adopted as the audit's reproducible rater if every item parses and its label kappa with the human reaches the lower end of the locked human~A label interval in label-audit-v2.json (the script reads it from there and records the decision); otherwise it is reported as tried and not adopted, as the decision model was (2026-09-30). That bar is lenient, since rater A agrees with the human less than with rater B, which is the reason to report all three pairs beside it. The items are the refined examples the audit sampled, read through refined_examples; an audited example a later refinement removed (the audit drew from corpus v2) is left out rather than rebuilt, listed by id, and every pair is read on the items that remain. The committed rubric differs from the raters' copy only in its worked example (invented here, a corpus item there).