The decision model as a third rater: well short of either model rater on the label¶
label-audit-v2-decision.json (Nimble, Ollama 0.35.0, digest 24e550a16a70; raters/C beside the
sheet). The same command that reproduces label-audit-v2.json with rater C added. Nimble's
answers are deterministic: six items re-asked returned the same choice and confidence.
| pair, label, 152 checked items | kappa | AC1 |
|---|---|---|
| human~B (the comparator) | 0.341 [0.172, 0.499] | 0.763 [0.687, 0.835] |
| human~C | 0.076 [-0.024, 0.179] | 0.631 [0.542, 0.717] |
Against the human, Nimble never returns partial or context_dependent where the human did (0 of
10 and 0 of 13 in the confusion table) and calls 13 of the human's 123 valid items
unrelated_rewrite. On outside_names its AC1 is higher than either model rater's but its kappa
is near zero: it answers "no" almost everywhere, and so does the human. Its agreement with A and
B over all 383 items is also low (label kappa 0.197 and 0.205).
Not adopted as a second labeller. The planned question was whether a local decision model agrees with the human about as well as the model raters do; on the label it does not, and the classes it misses are the two the audit exists to find. Its confidence does rank its own agreement (a one-off read on the checked items, not committed, so not quoted), which would fit triage (sending its least confident items to a stronger rater) rather than labelling; that is not pursued now.