Skip to content

Research log

The stopping rule is liberal at the K it stops at, so K is fixed from the pilot: 24 for OpenStack

The simulation of the registered rule (partition-sensitivity-openstack-stopping.json, commit 2cd4184): 1,000 null studies and 300 per power step at each run spread, the test window's 1,635 planned changes drawn from the v3 pilot, runs on admissible partitions built as the jobs build them, Algorithm 1 from K = 8 to 40. One-sided false positives against nominal 0.0125 (97.5%) and 0.025 (95%), mean runs, share at the cap, and the rate at which a second, independent aggregation differs by more than xi = 0.01:

  • in Corpus
  • 2 min read

The repeated-partition design on the development window: it stops at 16 runs, and H1 is inconclusive at +0.007

Twenty-two partition_run jobs on corpus-v3/openstack, the first 22 admissible partitions at training seeds 1 to 22 and N = 1,850 (jobs 208523 to 208530 pinned at 8ec4e12, 208535 to 208548 at 205ba2f; the two differ only by scripts/partition_pilot.py, which no job runs). Every outcome-neutral check passed and the apparatus held in all 22, and every run scored the same 501 examples: dedup-first left nothing to drop.

RQ2 is rebuilt on permitted hosts, and FedSAIL joins the defences

Decided with AJ ("item 3"). Every RQ2 measurement so far reads AOSP and Qt corpora fetched over REST before the robots.txt stop (2026-09-23): the codebase-family attribution, the aggregate attack and all four adapter-split defences. They are kept as internal pilots and nothing built on them is published. RQ2's publishable corpora are the permitted hosts: OpenStack, Wikimedia, and LineageOS, whose Android platform code stands in for AOSP's C, C++ and Java family. AOSP returns through the NoteDb route once its host's terms allow bulk fetching; Qt returns only with permission.

Candidate organizations beyond Qt and Chromium: Wikimedia, then LineageOS; the Linux Foundation hosts are too small

Qt and Chromium have not answered the requests for permission, and H2 runs only if both are admitted. Decided with AJ: the Stage 1 design names an ordered list of candidate organizations and the criteria that admit one (a host whose robots.txt and terms permit automated access, or written permission; the split criteria at the design's training size), so a host that never answers moves the study to the next candidate rather than stopping it. Registered-report guidance asks for inclusion criteria and the conditions under which units are replaced to be fixed in Stage 1.

OpenStack on corpus v3: seed effect 0.0077, detectable +0.0246 and +0.0210, and a dev pilot that passes on a different partition

Five single-seed placebo jobs at the half size (207303 to 207307, pinned at 8334d79) on corpus-v3/openstack, copied to TIGRIS with all 109 files' sha256 matched. The five roots are byte-identical under the same digest command as v2: 0ee3f74ab4a1b6e9. Every outcome-neutral check passed in every job. The halves equalize at 2,004 and are scored on 98 and 109 development changes (v2: 116 and 113).

  • in Corpus
  • 4 min read

Corpus v3: examples whose target the reviewer wrote are removed, since their share differs by organization and mostly grows between windows

Found while auditing Wikimedia's first eight built months for bot comments (none were found, and the build's service-account filter has removed nothing on any of the four hosts). What turned up instead is Gerrit's "Suggest edit": the reviewer's rewrite of the commented lines, fenced as a ```suggestion block in the comment (USER_SUGGESTION_START_PATTERN in Gerrit's polygerrit-ui/app/elements/shared/gr-comment/gr-comment.ts, read at c52ea5b). The prompt carries every comment, so on those examples the target is text the prompt already holds. Counted on the built examples (before refine):

Registered before Stage 1: H1 carries a file-type-matched supplementary estimand

Decided on the open question the language-mix ceiling raised: OpenStack's halves split 40% reStructuredText against 62% Python, so part of a half-split contrast could be a document-type difference. H1 now carries a supplementary estimand, as H2 carries one restricted to C++: the same contrast on one file suffix per admitted organization, the one both halves hold most of, meaning the largest smaller count across the two halves' training rows, ties by name (matched_suffix). Pooling the halves was rejected in review: a large docs-heavy half could make a suffix the small half barely has the pooled winner, and that cell would fail for want of changes. For OpenStack the rule gives .py (692 and 1,362 training rows; .rst has 871 and 313). The suffix is derived in code from the organization's split-criteria artifact, never passed in, and each half's test-window share of it is reported. Under ICH E9(R1) a restricted population is a different estimand, so it is reported beside H1 and decides nothing. The rule was registered after OpenStack's development-window pilot had been read, and reads only training-window suffix counts. It costs no training: the adapters' existing scores are read on the subset. cpp_supplement and file_type_supplement share one restricted-bootstrap helper, which builds each cell's clusters inside its error handling, as cpp_supplement did.

OpenStack on corpus v2 at the half size: the seed effect holds five seeds, and the dev-window H1 pilot is inconclusive at +0.013

The first GPU runs on corpus v2. The v2 OpenStack corpus was copied to TIGRIS (corpus-v2/openstack, every file's sha256 matched) and the placebo run at the half size under RUN_TAG=v2, five single-seed jobs (206166, 206171 to 206174), each building its halves into its own root so concurrent builds could not overwrite one another. The five roots are byte-identical, checked on the cluster with find . -type f ! -name '*.source.json' ! -name placebo.json -print0 | sort -z | xargs -0 sha256sum | sha256sum in each root: every one gives f49b745cfbde62c0. --corpus-digest records that value when the runs are merged; it is the caller's assertion and the merge does not recompute it. merge otherwise refuses runs whose corpus sources differ, and with the digest compares them with only the root removed, mode, windows and every count still compared. The halves equalize at 2,158 examples, of which 2,155 trained (three refused by the trainer), and are scored on 116 and 113 development changes.

Registered before Stage 1: the confirmatory cells are a rule over the admitted organizations

The granularity design had two fixed cell sets, with and without Chromium, and both put Qt in H1. Since 2026-09-26 an organization enters a confirmatory cell only from a host permitting automated access in writing, so Qt, Chromium and Wikimedia are each admitted or not independently, and two fixed sets no longer cover the cases. The cells are now built by one rule from the admitted set (design() in sphragis/experiment/decomposition.py), fixed here before any Wikimedia contrast exists: