Skip to content

Corpus

  • in Corpus
  • 9 min read

The review UIs are closed to crawlers and the git hosts are not: a NoteDb route, checked against the REST corpus

# research(2026-09). scripts/notedb_parity.py fetches every change NoteDb records as submitted in 2024-11 and 2025-01 on AOSP's platform/hardware/interfaces, over git and on every branch, builds them with the same build_from_change as the REST corpus, and compares the two into datasets/results/notedb-parity-aosp.json. The artifact was regenerated on 2026-09-25 from the branch as reviewed: the enumeration, field, example, rebase and timing figures below are from that run, and figures from earlier runs are dated where they appear. The first run's (2026-09-23, main only) are superseded.

  • in Corpus
  • 1 min read

Both corpus v2 manifests reproduce from the refined examples

The build and refine rule digests make a month built or refined under other code unreadable, but dedup and split sit under neither, so a change to either would leave the frozen windows intact and verify clean: the failure that let Qt verify at 10,695 while the pipeline yielded 10,692. verify --reproduce reruns dedup and split in memory from the refined examples and compares every window byte for byte with its frozen split file, over every window either side names, and the dedup counts with the manifest. Comparing ids alone would pass a dedup that kept a different copy of an id, and comparing only the rerun's windows would pass one dropped from the bounds (both found in review). On the corpus v2 data, OpenStack and Qt both reproduce byte for byte. The check takes minutes on Qt, so it is a flag rather than the default.

The audit's kappa is the prevalence paradox: AC1 0.853, and the raters part mostly on whether an item is valid

Rater A calls 311 of 383 items valid and rater B 323, so both raters' marginals sit on one label and chance agreement under kappa is high. That depresses kappa at high raw agreement (Feinstein and Cicchetti 1990), which is why kappa is reported beside raw agreement and a paradox-resistant coefficient rather than alone (James, LREC 2026, arXiv 2603.06865). Gwet's AC1 takes chance agreement over the whole five-label scale (Gwet 2008). Same items, same bootstrap seed, so the kappa intervals reproduce byte for byte (label-audit-v2.json, regenerated):

Label audit v2: two blind model raters agree at kappa 0.56, and validity does not differ between the units a contrast compares

The first audit's context-dependent class folded two questions together: whether the request can be understood from what the example shows, and whether the rewrite uses names the example does not show. A clear request answered with a project's own constant is a valid label that no prompt alone reproduces, which is house-style knowledge an adapter can learn, not label noise. Rubric version 2 asks the two separately (scripts/label_audit/rubric.md).

Correction: every successor is a rework, and corpus v2 is refrozen under the corrected rules

Retracted: the rebase-only-successor finding in the two entries above (0.34% of OpenStack's kept examples and 1.65% of Qt's in the audit, 20 and 189 in the v2 table). It was an artifact of the audit's own lookup, which keyed a change's revisions on its Change-Id. A cherry-pick carries the Change-Id to another branch, so a stable-branch copy that was only rebased lent its kind to the original. Resolved per change (by number, else by Change-Id, project and creation time together), every OpenStack successor and 11,444 of Qt's 11,448 are reworks; the other 4 are changes that cannot be told apart and are kept and counted. The rule stays as a guard, and now removes nothing. A review of the v2 branch caught it. Rebase edits carried inside a reworked successor remain the open question the audit entry names.

At the test window's size each H1 cell detects about three exact-match points, and absence is out of reach

scripts/decomposition_sensitivity.py simulates one H1 cell as the gate reads it: the placebo's own-against-sibling contrast on each half as the variance model, the test window's projected changes split between the halves in their dev-window proportion, five seeds at each organization's half-size seed effect, and the stratified crossed interval. Marginal power 0.928 per cell, so three independent cells pass together about four times in five. Generated from decomposition-sensitivity.json (100 trials a step, 1000 resamples).

Sibling-half leakage is below the registered threshold on the dev window

H2 would credit the organization with shared boilerplate if a sibling half's training data already held near-copies of the evaluated half's refinements. scripts/sibling_leakage.py assigns halves exactly as the placebo does and measures, for each half's dev window, the share of examples whose closest training example reaches a Jaccard threshold: from its own half's train window, its sibling's, and the foreign organization's. Generated from datasets/results/sibling-leakage.json.

  • in Corpus
  • 1 min read

The two probes now differ on purpose, and the seal is enforced where months are read

scripts/separability_over_time.py. The review noted that this probe reads every built month, dev included, where separability.py stops at train, and left it. The difference is right and was undocumented: this one asks whether the corpus moves over time, and a drift reading that stops two months early cannot see the end of its own series. It reads corpus text and no held-out predictions, so the dev window is described rather than spent. The docstring now says that instead of implying both probes follow one rule.

  • in Corpus
  • 1 min read

256 examples a client is not a third point on this corpus

At 256 examples, AOSP's system/core (204 usable examples) cannot fill one client, and the job refused to drop the source silently. Run anyway, AOSP would fall to two projects and four clients, and the project-level permutation from 84 arrangements to 28, with a floor of 1/28 rather than 1/84. That would be a different and weaker test on a thinner set, not a further point on the same curve, so it is not run. Long local training needs larger per-project corpora, which is what the ten-project AOSP rebuild is for.

  • in Corpus
  • 4 min read

The contamination battery separates the windows, on fifteen times the sample

datasets/results/contamination-openstack-6mo-with_context.partial.json. Six post-cutoff months (2024-10 to 2025-03) against the six-month control (2023-08 to 2024-01), built by one pipeline under one cutoff, scored on hunks with context on the base Qwen/Qwen2.5-Coder-7B. The job was still running guided completion when these were read: the membership scores are written before the generative half starts, which is the change made after job 145093 lost exactly this result to its wall.

  • in Corpus
  • 2 min read

Qt's ban lifts, and the corpus is complete

scripts/resume_when_allowed.sh probed Qt's Gerrit with five requests every twenty minutes from 20:29. Fourteen probes answered nothing. The fifteenth, at 01:45, answered all five, and the script built 2025-10 at one request a second, finishing 04:30: 826 examples, and Qt's thirteenth month. The snapshot had been on disk since the fetch that preceded the ban, so nothing was refetched; only the build, which needs a comments request per change and a diff request per comment, had been blocked. Total ban duration from the first refused handshake: about thirty-three hours.