Skip to content

Corpus

  • in Corpus
  • 2 min read

Backports of code older than the corpus: 19 of 4,006 OpenStack training examples are backports with no original in the corpus

backport-share.json (scripts/backport_share.py), metadata only. ROADMAP Plan H asked whether pre-cutoff code enters through cherry-picks: a change's window is its creation month, so a backport created inside the corpus can carry code first written before the checkpoint. A backport that keeps its original's Change-Id can be traced, and the examples keep no branch, so each one is read two ways. When every change with its (project, Change-Id) sits on a release, maintenance or deployment branch (as the hosts name them, from OpenStack's stable/ and StarlingX's r/stx to MediaWiki's REL1_xx, fundraising/REL1_xx and deploy/wmf/), its original is not in the corpus and may predate it. When any does, it may be such a backport, whose original is in the corpus and postdates the checkpoint with it.

  • in Corpus
  • 1 min read

The raw snapshots keep Gerrit account numbers outside _account_id; the examples do not

Found while checking whether reviewer overlap can be read from raw snapshots. scrub replaces every _account_id value and every address with a salted pseudonym, but two other places carry the raw Gerrit account number: attention_set and removed_from_attention_set are dictionaries keyed by it, and their reason text names it as <GERRIT_ACCOUNT_n> (lines with one: OpenStack 12,450, Wikimedia 19,134, AOSP 5,215, Qt 9,910 across the raw months on disk). Account numbers are public on each host and resolve to a person, so the raw snapshots are not pseudonymous in full. No refined or built example holds one (all four corpora checked), so nothing derived, committed or released is affected; the GitHub route's rows have no attention sets. The fix to scrub waits for the next build-rules change, since scrub.py is in BUILD_SOURCES and an edit stales every frozen month; cleaning the snapshots already on disk is a rewrite of raw data, the same decision as purging withdrawn changes, and is AJ's (ROADMAP Plan A). Until then the claim is that identities are pseudonymised in everything derived from the raw snapshots, not in the snapshots.

  • in Corpus
  • 2 min read

AI-assisted targets: 11 of OpenStack's 214 dev-window changes carry an AI trailer; Wikimedia none

ai-trailers.json (scripts/ai_trailers.py). OpenInfra's AI policy (approved 2025-07-08) asks for Assisted-By: or Generated-By: trailers on AI-assisted commits, and the raw snapshots carry no commit messages, so each trailer is a Gerrit message: search bounded by mergedbefore: the test window's start (no sealed change is requested), and each hit's merged commit message is read. A change is flagged when a trailer line's value names an AI tool or model; the windows are the corpus's own, refined examples deduplicated and split as the study loads them. Corrected before this entry was written: Gerrit's message: matches words anywhere, so only trailer lines count; most OpenStack Generated-By: lines name the community's scripts (1,132 changes, e.g. openstack/releases:tools/...), counted apart as tool-generated; and a tool name that is also a personal name (Devin, Junie, Cline) counts only where the value names a tool, never as a co-author. An independent review found the first denominators were the refined rows before dedup, and that the trend was read off examples clustered within changes; both are fixed here.

  • in Corpus
  • 5 min read

A GitHub collection route, and what live pull requests taught it

sphragis/corpus/github.py and github_api.py (spec docs/superpowers/specs/2026-10-01-github-route-design.md): pull requests shaped as Gerrit changes, so build, refine and dedup run on them unchanged and no frozen Gerrit month goes stale; the route's own digest (GITHUB_RULES) stales only GitHub months, failing closed when a GitHub snapshot lacks it. fetch --via github collects a registered organization-month; the sealed test window is refused before any request, as on the other routes.

  • in Corpus
  • 2 min read

Wikimedia's test window projects as a lower bound: the dev check fails by under-predicting

window-report-wikimedia-v3.json then project-windows-wikimedia-v3.json, on the v3 corpus. Train-into-dev leakage is 0.27% at Jaccard 0.7 (2 of 747 dev examples). The capture model, fitted on the training window (2,348 changes over 10 months, captured 0.974, 241.1 a month), predicts 359 dev changes against 448 observed: a miss of -19.9%, past the 15% tolerance. Wikimedia's arrivals rose over the year (raw changes a month in the refined corpus range from 156 to 319, the highest in the first dev month), so the dev window holds more than the training rate predicts.

  • in Corpus
  • 1 min read

Wikimedia's development-window pilot: K = 16, and H1 inconclusive at +0.0009

Twenty-two partition_run jobs on corpus-v3/wikimedia, the first 22 admissible partitions at training seeds 1 to 22 and N = 1,850 (jobs 210577 to 210600, pinned at ec8cc05). Every outcome-neutral check passed and the apparatus held in all 22; the largest leakage rate, own or sibling, is 0.40% against the registered 2%. Every run scored the same 711 examples (429 changes); none was dropped.

  • in Corpus
  • 2 min read

Planned before any run: a local decision model as a third blind label-audit rater

Exploratory, outside the registered pipeline. Ollama 0.35 serves decision models over a System One API (/v1/systemone): a choice question returns a probability per option and a yes/no question the probability of yes, in one pass, with no generated text. Nimble (Bespoke Labs, 9B, Qwen3.5-9B base, Apache 2.0) runs locally, so no corpus text leaves the machine; TypeSafe's hosted Jev, the API it follows, is not used, since its data terms are unpublished. The only published evaluation outside its vendors found a System One model's calibration model-specific, to be audited per model (Rafe and Das, arXiv:2609.24052).

  • in Corpus
  • 1 min read

The test window re-projected over its twelve months on v3: 2,018 OpenStack changes

project-windows-openstack-v3.json, from window-report-openstack-v3.json (both new; the projection now writes an artifact and reads the corpus it is given). v3's train window holds 1,645 changes over ten months, captured at 0.952, so 172.9 true changes a month; that rate predicts the dev window at 211 against an actual 214 (-1.3%, inside the 15% tolerance). Over the registered twelve months, fetched at the earliest month the horizon permits (2027-01, captured 0.973), the test window projects to 2,018 changes, replacing the ten-month 1,809 scaled to v3 (1,635). The simulation now reads its test size from this artifact (--projection), not a typed --size.

An audit of this log against its artifacts: nine conclusions in force are wrong or unsupported

A reader with no part in writing these entries went through the log oldest to newest, re-deriving figures from the committed artifacts, and checked the current docs and the Stage 1 draft against it. Two findings were re-checked by hand before anything was acted on (the leakage halt and the test size, below). What stands, by consequence:

  • in Corpus
  • 2 min read

The repeated-partition design on the development window: it stops at 16 runs, and H1 is inconclusive at +0.007

Twenty-two partition_run jobs on corpus-v3/openstack, the first 22 admissible partitions at training seeds 1 to 22 and N = 1,850 (jobs 208523 to 208530 pinned at 8ec4e12, 208535 to 208548 at 205ba2f; the two differ only by scripts/partition_pilot.py, which no job runs). Every outcome-neutral check passed and the apparatus held in all 22, and every run scored the same 501 examples: dedup-first left nothing to drop.

OpenStack on corpus v3: seed effect 0.0077, detectable +0.0246 and +0.0210, and a dev pilot that passes on a different partition

Five single-seed placebo jobs at the half size (207303 to 207307, pinned at 8334d79) on corpus-v3/openstack, copied to TIGRIS with all 109 files' sha256 matched. The five roots are byte-identical under the same digest command as v2: 0ee3f74ab4a1b6e9. Every outcome-neutral check passed in every job. The halves equalize at 2,004 and are scored on 98 and 109 development changes (v2: 116 and 113).

  • in Corpus
  • 4 min read

Corpus v3: examples whose target the reviewer wrote are removed, since their share differs by organization and mostly grows between windows

Found while auditing Wikimedia's first eight built months for bot comments (none were found, and the build's service-account filter has removed nothing on any of the four hosts). What turned up instead is Gerrit's "Suggest edit": the reviewer's rewrite of the commented lines, fenced as a ```suggestion block in the comment (USER_SUGGESTION_START_PATTERN in Gerrit's polygerrit-ui/app/elements/shared/gr-comment/gr-comment.ts, read at c52ea5b). The prompt carries every comment, so on those examples the target is text the prompt already holds. Counted on the built examples (before refine):

OpenStack on corpus v2 at the half size: the seed effect holds five seeds, and the dev-window H1 pilot is inconclusive at +0.013

The first GPU runs on corpus v2. The v2 OpenStack corpus was copied to TIGRIS (corpus-v2/openstack, every file's sha256 matched) and the placebo run at the half size under RUN_TAG=v2, five single-seed jobs (206166, 206171 to 206174), each building its halves into its own root so concurrent builds could not overwrite one another. The five roots are byte-identical, checked on the cluster with find . -type f ! -name '*.source.json' ! -name placebo.json -print0 | sort -z | xargs -0 sha256sum | sha256sum in each root: every one gives f49b745cfbde62c0. --corpus-digest records that value when the runs are merged; it is the caller's assertion and the merge does not recompute it. merge otherwise refuses runs whose corpus sources differ, and with the digest compares them with only the root removed, mode, windows and every count still compared. The halves equalize at 2,158 examples, of which 2,155 trained (three refused by the trainer), and are scored on 116 and 113 development changes.