Corpus v2: the audit's rules, applied at the source and to what was already built¶

The fixes are in the builder, so every corpus collected from here on is clean at collection, and in a new refine stage that brings corpora built before them to the same rules from their own examples and raw snapshots, with nothing refetched.

Bots are recognised from their source, not from a guessed list. Identity first: Gerrit tags service accounts SERVICE_USER, and the scrub used to replace every account object with its pseudonym alone, discarding that tag at ingestion; it now keeps it. Where a host does not expose the tag, and for corpora collected before, the fallback is each bot's own message templates: sphragis/corpus/automated/qt-sanity-bot.json holds all 101 complaints git-hooks/sanitize-commit posts, extracted from qt/qtrepotools at 309df8d5eefb by scripts/extract_bot_templates.py (spelling complaints pinned to the exact shape the code writes, so a reviewer's "a -> b?" is not taken for the bot); Qt's QUIP-23 integration posts one fixed message on files carrying a Qt-Security header; and flake8 output is posted inline on pyside/pyside-setup. The earlier prefix scan had missed the last two and most of the spelling complaints. Recognised in Qt's built examples: Qt Sanity Bot 1,347, flake8 lint output 338, Qt QUIP-23 security review integration 54; in OpenStack's, none. Templates match whole comments. Every recognised string was inspected, and a repeated-text queue of the rest, which a new bot would surface in, holds only reviewer language.

Refrozen as corpus v2 (the v1 manifests stay in git history, and every earlier result cites them):

Re-read under the registry rather than the prefix list: Qt's organizational contrast is +0.0182 [-0.0022, +0.0388] with 0.055 of its held-out examples removed, the same reading as before.

What else changed with it. Everything that trains or measures now reads refined examples through one loader (sphragis/corpus/load.py) that refuses a refinement made from other examples or under other rules; nine scripts had read the built files directly, and a fix that reached only the CLI would have left them on the old data. A placebo's halves are derived corpora carrying the rules version they were cut under. The rules version is a digest of the registries and the acknowledgement list, so it cannot fall behind them. make data-audit re-runs this audit on any corpus and writes data-audit-<org>.json. The scrub treats a chained address as one.

Not yet done: the pilot adapters were trained on v1, including the bot examples; retraining on v2 waits on Qt's answer about access, since training is where the Qt data is used again. RQ2's project corpora are rebuilt from v2 by project_corpora.py before its next run.

Leakage re-measured on v2 (sibling-leakage.json, window-report-*.json regenerated). OpenStack train into dev at Jaccard 0.7 is 0.0106, unchanged. Sibling-half rates at Jaccard 0.7:

All below 2%. openstack-a's sibling rate exceeds its own half's, so the threshold, not an ordering between the own half and the sibling, is the property to rely on.