Skip to content

Research log

The dated record of what was built, measured, found and corrected, newest first. Withdrawn and superseded readings stay, marked, so every correction can be audited. Browse by month or by category in the sidebar.

The GitHub route's digest restored; network-bound digests pinned

# research(2026-10). A comment edit to sphragis/corpus/github.py in #107 (2026-10-08) moved GITHUB_RULES from 56b5374b852d17d7, the digest every GitHub month records, to f89b7378b3675611, so the loader on main refused all of Apache's, LLVM's and .NET's months as "fetched under other GitHub route rules". The collector runs an older checkout and never saw it. github.py is restored to its 6804898 bytes, which brings the digest back; the comment's reading of GitHub's Acceptable Use Policies is the older one, and the 2026-10-08 entry above stays the current word. The loader then reads every Apache, LLVM and .NET month. It is the second time the route's digest moved under collected months (a code change in #101, restored in #102, was the first), so tests/unit/corpus/test_rule_pins.py pins BUILD_RULES and GITHUB_RULES to the digests the collected months record, and FETCH_RULES to the code the git route's first fetch will record; a deliberate rules change updates its pin in the same commit as the rebuild it implies. RULES_VERSION stays unpinned, since a label change is re-refined from disk.

  • in Corpus
  • 2 min read

Qt's development pilot on its granted corpus: H1 supported on likelihood, K = 10

# research(2026-10). Qt's 22 development runs, the k-th on the k-th admissible partition at training seed k and N = 1,850, each chained partition_run -> decoder_check -> likelihood_score at the registered decoder (TIGRIS 298699 to 298764, pinned at faa8e8d), read by partition_pilot.py in admissible order (partition-pilot-qt-likelihood.json, partition-pilot-qt-rp1.0.json):

  • in Corpus
  • 2 min read

Qt's corpus under its grant: the count probe holds, the split qualifies, 40 partitions are admissible

# research(2026-10). Qt's months 2024-10 to 2025-10, fetched under the 2026-10-15 grant from the announced address, were built with build --fewer-requests (started at 13bd2a4, before the probe gate existed), then refined, deduplicated, split, frozen and verified (2026-10-16, 23:34), and backed up. The probe the 2026-10-15 entry made a precondition then ran from committed code (comment-count-probe-qt.json, from 8.47.98.37): none of the 300 zero-count rows it asked carried a comment and every control did, so the share of skipped changes with comments is bounded at 0.0099, under COUNT_PROBE_BOUND, and the build stands without a rebuild.

Chromium grants the REST API: its pool becomes H2's foreign source

# research(2026-10). Chromium's infra-dev list answered the narrowed request of 2026-10-15, about 2,000 merged changes with their inline review comments created 2024-11 to 2025-08 from one client at no more than one request a second, on 2026-10-15 (18:28 PDT), from a Chrome Infra address: "There are no concerns with hitting Gerrit's REST API at such a rate." The thread is titled for chromium-review and android-review, and AJ reads the answer as covering both. The request named NoteDb over git; the answer names the REST API, which is the route used here. That scope is a foreign pool, not Chromium as an evaluated organization, whose full corpus stays blocked as on 2026-10-14 (chromium/src's 18,509 merged changes a month against the host's 10,000-result cap).

  • in Corpus
  • 2 min read

Qt's fetch under its grant, and the comment-count probe its build waits on

Qt's fetch under its 2026-10-15 grant finished the same day: the pilot, train and development months, 2024-10 to 2025-10, from the collector at the announced address, at 2 s, one client, with no stop. A first probe asked Qt for the comments of 60 changes its snapshots record at total_comment_count 0 and of 10 above zero: all 60 returned none and all 10 returned some. Sixty empty answers bound the share of zero-count changes that carry comments only at about 5% (95%, one-sided), too loose for a skip applied to about half of Qt's changes, and the script that wrote that result was edited after it ran, so the result is not kept.

s split by token class: OpenStack's development contrast is not vocabulary alone, and it sits in the edit

# research(2026-10). Review panel item 5 asked whether s measures an organization's vocabulary rather than its conventions, since a sibling adapter has seen the organization's names and a foreign one has not. likelihood_score.py now classes each target token as identifier or other and as changed or copied, and stores each class's contribution to the mean per token (sphragis/experiment/tokens.py): identifiers are Pygments' names other than the language's builtins, a token touching one counting as one, so other is conservative; only the languages these corpora hold are lexed, with the suffixes Pygments misreads pinned, and anything else reads as text; changed or copied comes from aligning lines and then, within replaced lines, words and symbols with the reviewed hunk. The contributions add back to s within fp32 rounding: identifier, other and end, or changed, copied and end. This replaces the Stage 1 draft's plan to reread both hypotheses on identifier-renamed references: renaming moves every arm off the distribution it was trained on and removes naming conventions, which are house style too.

A REST build can ask only what the host has not answered

A REST build asks the host for each change's comments and, for each anchored comment, a diff, and under Qt's granted interval those requests set how long its build takes. build --fewer-requests asks less: a change whose snapshot records total_comment_count 0 (Gerrit counts every inline comment across a change's patch sets there) is not built and is counted as uncommented_by_count in the month's drops, and each diff is asked once per change however many comments sit on that file and patch set (a failed request is not remembered). The flag is off by default, so every build without it, OpenStack's and Wikimedia's included, is as before; a month built with it says so by carrying the uncommented_by_count key, even at zero.

  • in Corpus
  • 2 min read

Wikimedia's corpus v3 frozen; the dataset's funnel published

The 2026-10-01 entry calls the OpenStack and Wikimedia v3 corpora frozen, but only OpenStack's copy (corpus-v3/openstack on TIGRIS) carried the freeze manifest and splits; Wikimedia's held its raw, built and refined months alone. No result depends on it, since every partition job deduplicates the organization itself (placebo_corpus.py --dedup-org). The dedup, split, freeze and verify stages were run on a copy of corpus-v3/wikimedia at 2dad2a1: they keep 5,513 of 5,699 refined examples (117 exact copies and 69 near duplicates removed), split as 582 pilot, 4,184 train and 747 development, which is window-report-wikimedia-v3.json exactly, and verify passes. The manifest and splits were added to corpus-v3/wikimedia beside the files already there, byte for byte.

Planned before any planted quote-style result is read: the exchange rate on a code-level convention

# research(2026-10). δ is carried from exact match to s at the rate of the adapters' gain over the base model on each scale (0.4372 / 0.2384 over 92 own-half development arms, 2026-10-09), and most of that gain is the adapter learning the task's output format, which says little about the rate for small differences between adapters (Stage 1 review panel, item 15). This measures the rate on a convention of the kind the study asks about, fixed here before any run:

  • in Corpus
  • 2 min read

The label audit read on OpenStack's items alone

The audit sampled OpenStack and Qt alike, and the Stage 1 text quoted agreement pooled over all 383 items, so a published figure rested partly on Qt's review records, fetched before Qt granted permission. scripts/label_audit_by_org.py reads the committed agreement report again on one organization's items through the same function as the pooled reading (label_audit_agreement.readings), so every pair is read: the five labels, valid against the rest, outside names, the model raters on the human's items, the human pairs without the answers reported as slips, and the halves of the human check. The report keys labels by example id in sorted neutral-id order and slips by neutral id; the ids are rebuilt from that order, and the whole pooled reading is recomputed from them and refused unless every pair reproduces the report (all do, to 1e-12). Valid shares are the report's own per-organization rates.

Qt grants the REST API in writing; its host joins REST_PERMITTED

Qt's Gerrit administrators answered the 2026-10-14 request by email on 2026-10-15 (QTQAINFRA-8064): requests at a 2-second interval are fine, an identifying User-Agent works for them, and they asked for the address the requests come from, which was sent the same day (the collector's own address; a change is announced before collection resumes). If their automated systems block the client, they unblock it on request. codereview.qt-project.org is now in REST_PERMITTED at those terms: 2 s between requests and one client machine-wide. The 15-minute quiet period after any 5xx is ours, as long as Wikimedia's, since Qt's firewall once outlasted 25 minutes of silence (pacing.Pacer). robots.txt still reads Disallow: /; the written grant is what opens the host.

The matched foreign draw implemented

holdout.matched_foreign_draw builds item 3 of the H2 plan: for one run's sibling draw it takes as many in-language rows from the foreign pool as the sibling holds and fills the rest from the pool's other rows, each stratum shuffled by the run's training seed and the sibling it is matched to (a refinement of item 3's "the run's training seed", so a run's two foreign adapters get independent draws). A pool that repeats an example id is refused, since repeats would count as capacity. A pool that cannot cover a draw is refused, never trained smaller, since that is a source that does not qualify. The partition jobs call it once a source's maintainers grant permission and its pool is built; no foreign draw has been made.