The human check: 77% raw agreement with rater A, below the model pair, and outside names is where the human and the raters part¶

Read under the rules registered above and on 2026-09-24, on the answers as locked (label-audit-v2.json, regenerated; the model raters' labels and all-item pairs reproduce unchanged). Every human pair a reported slip touches is also read without that item. The model pair is re-read on the 152 checked items, the comparator for every human row:

pair	items	raw	kappa	AC1
A~B, label, all items	383	0.864	0.557 [0.454, 0.652]	0.853 [0.813, 0.890]
A~B, label, checked items	152	0.888	0.654 [0.512, 0.782]	0.878 [0.818, 0.930]
human~A, label	152	0.770	0.300 [0.154, 0.440]	0.749 [0.669, 0.823]
human~A, label, without slips	143	0.818	0.368 [0.194, 0.530]	0.804 [0.731, 0.874]
human~A, valid or not	152	0.822	0.417 [0.226, 0.590]	0.745 [0.637, 0.837]
human~A, valid or not, without slips	143	0.860	0.482 [0.274, 0.669]	0.808 [0.716, 0.889]
human~A, outside names	152	0.816	0.092 [-0.072, 0.280]	0.770 [0.670, 0.855]
human~A, outside names, without slips	139	0.892	0.229 [-0.052, 0.492]	0.875 [0.802, 0.935]
human~B, label	152	0.783	0.341 [0.172, 0.499]	0.763 [0.687, 0.835]
human~B, label, without slips	143	0.818	0.381 [0.190, 0.553]	0.804 [0.732, 0.872]
human~B, valid or not	152	0.809	0.374 [0.180, 0.550]	0.726 [0.613, 0.821]
human~B, valid or not, without slips	143	0.839	0.414 [0.199, 0.605]	0.778 [0.677, 0.865]
human~B, outside names	152	0.789	0.112 [-0.048, 0.283]	0.727 [0.614, 0.819]
human~B, outside names, without slips	139	0.856	0.217 [-0.008, 0.435]	0.824 [0.733, 0.897]
human~A, label, first half	76	0.763	0.270 [0.069, 0.471]	0.742 [0.632, 0.846]
human~A, label, second half	76	0.776	0.330 [0.117, 0.533]	0.756 [0.646, 0.860]

Below the model pair. On the same items the two models reach AC1 0.878; the human reaches 0.749 with rater A and 0.763 with rater B, and the intervals barely overlap. Kappa sits lower throughout because one label dominates (the prevalence paradox, 2026-09-23).
No detectable rise toward rater A, at low power. Human~A does not rise from the first half to the second beyond noise, but each half's interval is wide. Agreeing with rater B as often as with rater A is weak evidence on its own, since the two models agree on 88.8% of these items. On the 17 items where they disagree, the human sides with A on 4, with B on 6 and with neither on 7; by half, 3, 6 and 1 of 10, then 1, 0 and 6 of 7. Too few items to read a trend.
The valid share is consistent across readers. On the checked items the human calls 123 of 152 valid, rater A 124 and rater B 124, while the item-level agreement on valid or not is only moderate. The disagreements run both ways: 10 items the human called valid and rater A partial, 8 the human called context-dependent and rater A valid.
Outside names is the weak question. The human flags 10 items, rater A 24 and rater B 30, and kappa on this question does not exclude zero, with or without slips. It carries 13 of the checker's reported slips against 9 on the label.

What this supports: the valid share the corpus reports is consistent with a human reading, within a 152-item subsample checked after each reveal. It does not support substituting the model labels for a human's item by item, on the finer labels or on outside names. The checker reported finding the reviewer's request and the code change harder to follow than rater A did; which reader is right on a disputed item is not measured here.