Skip to content

October 2026

Checkpoints pinned to the revisions every run used

Nothing passed a revision when loading a checkpoint, so every load read the Hub's main, and both 7B repositories changed their config and tokenizer files on 2024-11-18, after publication. The cluster's cache held one revision of each for every run to date: Qwen2.5-Coder-7B-Instruct c03e6d3 (README-only changes after 2024-11-18) and Qwen2.5-Coder-7B 0396a76; the 1.5B dev model is pinned at its current 2e1fd39. MODEL_REVISIONS in sphragis/experiment/model.py now passes them to every load, and a checkpoint without one refuses to load. Past runs are unaffected: they read these revisions.

Reviewers are shared across halves; the reviewer analysis reads exposure, not a shared-reviewer split

reviewer-overlap.json (scripts/reviewer_overlap.py), metadata only, no outcome read. A change's reviewers are the pseudonymous accounts in its attention-set history other than its owner, change-level rather than the author of the example's own comment (the built examples do not keep it). Halves are the registered largest-first assignment; windows the corpus's own.

  • in Corpus
  • 1 min read

The raw snapshots keep Gerrit account numbers outside _account_id; the examples do not

Found while checking whether reviewer overlap can be read from raw snapshots. scrub replaces every _account_id value and every address with a salted pseudonym, but two other places carry the raw Gerrit account number: attention_set and removed_from_attention_set are dictionaries keyed by it, and their reason text names it as <GERRIT_ACCOUNT_n> (lines with one: OpenStack 12,450, Wikimedia 19,134, AOSP 5,215, Qt 9,910 across the raw months on disk). Account numbers are public on each host and resolve to a person, so the raw snapshots are not pseudonymous in full. No refined or built example holds one (all four corpora checked), so nothing derived, committed or released is affected; the GitHub route's rows have no attention sets. The fix to scrub waits for the next build-rules change, since scrub.py is in BUILD_SOURCES and an edit stales every frozen month; cleaning the snapshots already on disk is a rewrite of raw data, the same decision as purging withdrawn changes, and is AJ's (ROADMAP Plan A). Until then the claim is that identities are pseudonymised in everything derived from the raw snapshots, not in the snapshots.

  • in Corpus
  • 2 min read

AI-assisted targets: 11 of OpenStack's 214 dev-window changes carry an AI trailer; Wikimedia none

ai-trailers.json (scripts/ai_trailers.py). OpenInfra's AI policy (approved 2025-07-08) asks for Assisted-By: or Generated-By: trailers on AI-assisted commits, and the raw snapshots carry no commit messages, so each trailer is a Gerrit message: search bounded by mergedbefore: the test window's start (no sealed change is requested), and each hit's merged commit message is read. A change is flagged when a trailer line's value names an AI tool or model; the windows are the corpus's own, refined examples deduplicated and split as the study loads them. Corrected before this entry was written: Gerrit's message: matches words anywhere, so only trailer lines count; most OpenStack Generated-By: lines name the community's scripts (1,132 changes, e.g. openstack/releases:tools/...), counted apart as tool-generated; and a tool name that is also a personal name (Devin, Junie, Cline) counts only where the value names a tool, never as a co-author. An independent review found the first denominators were the refined rows before dedup, and that the trend was read off examples clustered within changes; both are fixed here.

Hardening against October 2026 practice: what the study already leads on, and Plan H

Five literature sweeps (inference and design; evaluation and data; contamination; federated LoRA privacy; conventions and fine-tuning), each paper opened on arXiv, DOI or the venue page; most were read at abstract level, the full text where a claim below rests on it. Decided as ROADMAP Plan H. Claims about this repository were checked against the code before being written here.

GitHub's own conversion on OpenJDK 2024-11: 0.234 [0.183, 0.293], and Grafana drops to borderline for RQ2

Read as planned above (github-conversion-openjdk.json). The route listed 850 integrated PRs, all counted (none failed, none gone): 307 carried a reviewer's inline comment, 525 examples were built and 329 survived refine, which removed 196 as suggested edits, the reviewer having written the target (a larger share than on the Gerrit corpora). Against 1,404 reviewer threads under the sizing's rule, the conversion is 0.234, 95% interval [0.183, 0.293], inside the Gerrit range it replaces (0.217 to 0.282) at the centre and wider at both ends. All 19 of the sizing's draws in the month recount to the same thread counts, so the rule and the data agree. Nine PRs have more than one page of comments and are undercounted, as in the sizing, which errs toward too few threads and so a higher rate. refine leaves every GitHub example's successor kind unchecked, by design: GitHub has no Gerrit revision kind, and the route's own upstream guard does that job.

The exploratory second model family is Llama-3.1-8B, not Qwen3-8B

The registered model's contamination argument is that the corpus (from 2024-10) postdates the checkpoint's publication. The exploratory second family, an H1 dev-window replication outside the registered cells (ROADMAP Plan F), was listed as Qwen3-8B; Alibaba states no cutoff for it, the model reports early 2025, which falls inside the training window, and it was published in 2025. A sweep of current open-weight families (October 2026) against two requirements, open weights (RQ1 trains adapters, RQ2 reads their updates) and a checkpoint published before the corpus:

Planned before the pilot month is read: GitHub's own thread-to-example conversion

The sizing's example projections borrow the Gerrit conversion, and the two denominators differ (above), so GitHub's conversion is measured on a built pilot month before any GitHub organization is admitted. Method (scripts/github_conversion.py): every PR the route lists for the month, each counted with the sizing's own per-PR rule (pr_threads, moved out of github_sizing.py so both read one definition); the numerator is the refined examples the route built from those PRs. A PR the sizing scores zero (bot or AI author) keeps its examples in the numerator, so the rate times the sizing's thread total projects the examples the route would build. A PR the route failed to collect, or that GitHub no longer answers for at the census, counts on neither side, since its examples are unknown; one withdrawn at collection keeps its threads, as the sizing counted them. The script refuses when the listing differs from the snapshot's own count, when an example comes from an unlisted PR, when failed PRs go unnamed, or when the month has no reviewer threads. The interval is a 95% percentile bootstrap over PRs of the ratio. The sizing's own draws that fall in the month are counted again by the census and their agreement reported, a check on the rule and on the data since the draws were taken.

  • in Corpus
  • 5 min read

A GitHub collection route, and what live pull requests taught it

sphragis/corpus/github.py and github_api.py (spec docs/superpowers/specs/2026-10-01-github-route-design.md): pull requests shaped as Gerrit changes, so build, refine and dedup run on them unchanged and no frozen Gerrit month goes stale; the route's own digest (GITHUB_RULES) stales only GitHub months, failing closed when a GitHub snapshot lacks it. fetch --via github collects a registered organization-month; the sealed test window is refused before any request, as on the other routes.

The confirmation pass: Apache and LLVM qualify for an H1 cell, .NET and Grafana for RQ2, OpenJDK and HashiCorp are borderline

github-sizing-report.json, from the 200 draws an organization planned above (github-sizing-draws-<org>.jsonl). The design checks out: the same estimator applied to the number of PRs recovers the counts the search API reports to within 1% to 7% (Apache 108,600 against 107,888; HashiCorp 24,531 against 22,870, the widest). No draw hit the search frame cap. The thread-to-example conversion read from the built corpora is 0.217 (Wikimedia) to 0.282 (OpenStack); the example range below is the interval's low end at the low rate to its high end at the high rate.

Planned before the confirmation pass is read: GitHub organizations as candidates beyond Qt and Chromium

AJ asked for six or more organizations, so that RQ2 has a federation of several clients and RQ1 more than two cells. The permitted Gerrit hosts give OpenStack and Wikimedia; LineageOS yields 458 examples from about 29,800 merged changes (its reviewers rarely comment inline). GitHub's Acceptable Use Policies allow research use of public information when the resulting publications are open access; identities are pseudonymised at ingestion as on Gerrit.

LineageOS's 57 dropped changes were withdrawn from the host, not failed requests

LineageOS 2025-10 dropped 57 changes as comment_error, and a rebuild of that month dropped exactly 57 again with every other count identical, so the drop was not transient. A one-request-per-change probe of the month (stopped at 1,562 of 4,161 changes, to spare a host then returning 502s) found timeouts and 502s, which the client retries five times and which rarely survive that, and 404s, which it does not retry. Two of the 404s were checked by hand: the change itself now answers 404. They were public when the snapshot listed them (fetched 2026-09-29) and were deleted or made private since.

The planted check passes 10 of 10 on Wikimedia; the planted run's base arm is left out of it

The ten runs planned above (jobs 211277 to 211286, pinned at a43f7af), rq1-partition-wikimedia-*-n1850-plant0.5.json. All ten pass, every half's lower bound above zero (least +0.036), per-half estimates from +0.087 to +0.143, projected pass probability at least 0.9994 per run (0.9999 on average). By the rule fixed before the runs, check 5 is registered for Wikimedia as for OpenStack, at a half.

  • in Corpus
  • 2 min read

Wikimedia's test window projects as a lower bound: the dev check fails by under-predicting

window-report-wikimedia-v3.json then project-windows-wikimedia-v3.json, on the v3 corpus. Train-into-dev leakage is 0.27% at Jaccard 0.7 (2 of 747 dev examples). The capture model, fitted on the training window (2,348 changes over 10 months, captured 0.974, 241.1 a month), predicts 359 dev changes against 448 observed: a miss of -19.9%, past the 15% tolerance. Wikimedia's arrivals rose over the year (raw changes a month in the refined corpus range from 156 to 319, the highest in the first dev month), so the dev window holds more than the training rate predicts.

  • in Corpus
  • 1 min read

Wikimedia's development-window pilot: K = 16, and H1 inconclusive at +0.0009

Twenty-two partition_run jobs on corpus-v3/wikimedia, the first 22 admissible partitions at training seeds 1 to 22 and N = 1,850 (jobs 210577 to 210600, pinned at ec8cc05). Every outcome-neutral check passed and the apparatus held in all 22; the largest leakage rate, own or sibling, is 0.40% against the registered 2%. Every run scored the same 711 examples (429 changes); none was dropped.