Skip to content

Apparatus

.NET's own conversion: 0.241 [0.212, 0.272]; .NET stays RQ2-size and borderline for H1

github-conversion-dotnet.json. .NET's 2024-11 lists 4,114 PRs, all counted: 1,633 were opened by a bot or agent the route recognises and not collected (1,739 under the sizing's wider name rule, which counts them zero), 737 carried a reviewer's inline comment, 1,098 examples were built and 489 survived refine, which removed 609 as suggested edits. Suggestion threads are 1,078 beside 2,026 other reviewer threads, the largest share of the three pilots, so .NET reviewers write the fix themselves more often than OpenJDK's or LLVM's. Against those 2,026 threads the conversion is 0.241, 95% interval [0.212, 0.272]; all 15 of the sizing's draws in the month recount the same. On its own pilot .NET projects 2,398 to 19,122 examples, so it reaches RQ2 and stays borderline for an H1 cell; the three measured intervals all lie inside OpenJDK's, so the borrowed span and the other classes are unchanged (github-sizing-report.json).

LLVM's own conversion: 0.255 [0.231, 0.280]; LLVM reaches H1 and RQ2 on its own pilot

github-conversion-llvm.json, read as planned (2026-10-01). LLVM's 2024-11 lists 2,710 PRs, all counted (none failed or gone): 1,069 carried a reviewer's inline comment, 1,330 examples were built and 989 survived refine (340 removed as suggested edits). Against 3,872 reviewer threads under the sizing's rule the conversion is 0.255, 95% interval [0.231, 0.280], tighter than OpenJDK's on three times the threads, and all 19 of the sizing's draws in the month recount the same. LLVM's month was collected after the retry-on-resume fix, so its github_rules digest differs from OpenJDK's; no PR in either month failed, so the fix changed nothing in either.

Checkpoints pinned to the revisions every run used

Nothing passed a revision when loading a checkpoint, so every load read the Hub's main, and both 7B repositories changed their config and tokenizer files on 2024-11-18, after publication. The cluster's cache held one revision of each for every run to date: Qwen2.5-Coder-7B-Instruct c03e6d3 (README-only changes after 2024-11-18) and Qwen2.5-Coder-7B 0396a76; the 1.5B dev model is pinned at its current 2e1fd39. MODEL_REVISIONS in sphragis/experiment/model.py now passes them to every load, and a checkpoint without one refuses to load. Past runs are unaffected: they read these revisions.

Reviewers are shared across halves; the reviewer analysis reads exposure, not a shared-reviewer split

reviewer-overlap.json (scripts/reviewer_overlap.py), metadata only, no outcome read. A change's reviewers are the pseudonymous accounts in its attention-set history other than its owner, change-level rather than the author of the example's own comment (the built examples do not keep it). Halves are the registered largest-first assignment; windows the corpus's own.

Hardening against October 2026 practice: what the study already leads on, and Plan H

Five literature sweeps (inference and design; evaluation and data; contamination; federated LoRA privacy; conventions and fine-tuning), each paper opened on arXiv, DOI or the venue page; most were read at abstract level, the full text where a claim below rests on it. Decided as ROADMAP Plan H. Claims about this repository were checked against the code before being written here.

GitHub's own conversion on OpenJDK 2024-11: 0.234 [0.183, 0.293], and Grafana drops to borderline for RQ2

Read as planned above (github-conversion-openjdk.json). The route listed 850 integrated PRs, all counted (none failed, none gone): 307 carried a reviewer's inline comment, 525 examples were built and 329 survived refine, which removed 196 as suggested edits, the reviewer having written the target (a larger share than on the Gerrit corpora). Against 1,404 reviewer threads under the sizing's rule, the conversion is 0.234, 95% interval [0.183, 0.293], inside the Gerrit range it replaces (0.217 to 0.282) at the centre and wider at both ends. All 19 of the sizing's draws in the month recount to the same thread counts, so the rule and the data agree. Nine PRs have more than one page of comments and are undercounted, as in the sizing, which errs toward too few threads and so a higher rate. refine leaves every GitHub example's successor kind unchecked, by design: GitHub has no Gerrit revision kind, and the route's own upstream guard does that job.

The exploratory second model family is Llama-3.1-8B, not Qwen3-8B

The registered model's contamination argument is that the corpus (from 2024-10) postdates the checkpoint's publication. The exploratory second family, an H1 dev-window replication outside the registered cells (ROADMAP Plan F), was listed as Qwen3-8B; Alibaba states no cutoff for it, the model reports early 2025, which falls inside the training window, and it was published in 2025. A sweep of current open-weight families (October 2026) against two requirements, open weights (RQ1 trains adapters, RQ2 reads their updates) and a checkpoint published before the corpus:

Planned before the pilot month is read: GitHub's own thread-to-example conversion

The sizing's example projections borrow the Gerrit conversion, and the two denominators differ (above), so GitHub's conversion is measured on a built pilot month before any GitHub organization is admitted. Method (scripts/github_conversion.py): every PR the route lists for the month, each counted with the sizing's own per-PR rule (pr_threads, moved out of github_sizing.py so both read one definition); the numerator is the refined examples the route built from those PRs. A PR the sizing scores zero (bot or AI author) keeps its examples in the numerator, so the rate times the sizing's thread total projects the examples the route would build. A PR the route failed to collect, or that GitHub no longer answers for at the census, counts on neither side, since its examples are unknown; one withdrawn at collection keeps its threads, as the sizing counted them. The script refuses when the listing differs from the snapshot's own count, when an example comes from an unlisted PR, when failed PRs go unnamed, or when the month has no reviewer threads. The interval is a 95% percentile bootstrap over PRs of the ratio. The sizing's own draws that fall in the month are counted again by the census and their agreement reported, a check on the rule and on the data since the draws were taken.

The confirmation pass: Apache and LLVM qualify for an H1 cell, .NET and Grafana for RQ2, OpenJDK and HashiCorp are borderline

github-sizing-report.json, from the 200 draws an organization planned above (github-sizing-draws-<org>.jsonl). The design checks out: the same estimator applied to the number of PRs recovers the counts the search API reports to within 1% to 7% (Apache 108,600 against 107,888; HashiCorp 24,531 against 22,870, the widest). No draw hit the search frame cap. The thread-to-example conversion read from the built corpora is 0.217 (Wikimedia) to 0.282 (OpenStack); the example range below is the interval's low end at the low rate to its high end at the high rate.

Planned before the confirmation pass is read: GitHub organizations as candidates beyond Qt and Chromium

AJ asked for six or more organizations, so that RQ2 has a federation of several clients and RQ1 more than two cells. The permitted Gerrit hosts give OpenStack and Wikimedia; LineageOS yields 458 examples from about 29,800 merged changes (its reviewers rarely comment inline). GitHub's Acceptable Use Policies allow research use of public information when the resulting publications are open access; identities are pseudonymised at ingestion as on Gerrit.

The planted check passes 10 of 10 on Wikimedia; the planted run's base arm is left out of it

The ten runs planned above (jobs 211277 to 211286, pinned at a43f7af), rq1-partition-wikimedia-*-n1850-plant0.5.json. All ten pass, every half's lower bound above zero (least +0.036), per-half estimates from +0.087 to +0.143, projected pass probability at least 0.9994 per run (0.9999 on average). By the rule fixed before the runs, check 5 is registered for Wikimedia as for OpenStack, at a half.

The planted check at a half passes 10 of 10 runs: registered at a half, read on the planted run only

The ten runs planned above, repeated at PLANT=0.5 (jobs 210234 to 210243, pinned at b05a2d8), rq1-partition-openstack-*-n1850-plant0.5.json. All ten pass. Every half's lower bound is above zero (least +0.017); per-half estimates run from +0.053 to +0.142. The pass probability projected by the logged rule (normal per half, interval half-width over 1.96 as the standard error, product over halves) is at least 0.9989 for every run and 0.9999 on average. Both conditions of the rule hold, so check 5 is registered with the plant at a half: neutral.planted_convention, which reads the plant's strength and window from the run's corpus path and refuses any other, and partition_pilot.py --planted, which halts the cell when it fails.