Skip to content

Research log

Registered before Stage 1: the confirmatory design rests only on hosts that permit automated access

The test window is collected fresh after in-principle acceptance, on every organization. On 2026-09-26 neither permission request had an answer: Qt's ticket QTQAINFRA-8064 is open with no comment, and nothing has come back from Chromium's infra-dev list. codereview.qt-project.org and code.qt.io both serve Disallow: /, so the fallback registered on 2026-09-22 (H1 on OpenStack and Qt) still depends on a permission that may not come: Qt's test window cannot be collected without it. The Chromium deadline covered H2 and left that dependency unstated.

The human check: 77% raw agreement with rater A, below the model pair, and outside names is where the human and the raters part

Read under the rules registered above and on 2026-09-24, on the answers as locked (label-audit-v2.json, regenerated; the model raters' labels and all-item pairs reproduce unchanged). Every human pair a reported slip touches is also read without that item. The model pair is re-read on the 152 checked items, the comparator for every human row:

Registered before the human's figures: the check stops at 152 items, and its halves are read apart

The human checked item-002 to item-153 between 2026-09-24 and 2026-09-26, 152 items past the rubric's worked example. The check stops there, fixed before any human-to-model figure is computed. The page shows items in the batches' seeded order, so the 152 are a random sample of the 383, and a stop chosen after seeing agreement would be optional stopping.

Registered: negative readings are bounded by each cell's detectable effect, and rebase edits get a sensitivity analysis

Negative readings. The proposal of 2026-09-23 is adopted. A cell reads bounded when its interval's upper bound lies below the effect the sensitivity analysis says that cell detects at the Holm level it is read at; the gate reads the bounds from decomposition-sensitivity.json (detectable_effects) and refuses a confirmatory cell without one at every level. The SESOI band is reported beside every cell and decides nothing. Justification: Lakens, Scheel and Isager (2018) list the effect a study was designed to detect among the bases for an equivalence bound, and Lakens (2022) asks a design to say which effects it is informative about when it cannot resolve the SESOI. The bound is reachable where absence was not: the regenerated sensitivity artifact records null_reads_bounded per cell and level beside null_reads_absent: a true null reads bounded 0.97 of the time in three of the four H1 cells and 0.94 in Qt's at 95%, against 0.000 absent in all four; every other figure in the artifact reproduced exactly. The bounds are recomputed at N on corpus v2 before the seal opens. A reversed effect now reads bounded, since H1 is directional.

Registered before the human's figures: a reported slip stays as locked

The check page locks the human's label and outside-names answer before it reveals rater A's. A checker can notice a mis-click only after the reveal, and a correction made after seeing the rater is no longer blind, so it does not replace the locked answer. The rule, fixed before any human agreement is computed: every human figure is read on the answers as locked, and each pair a reported slip touches is also reported without that item (human_slips and the without slips pairs). Slips are marked on the page beside the locked answer and saved with the check; label_audit_agreement.py reads them from there, and --slip ITEM:FIELD adds one by hand.

  • in Corpus
  • 9 min read

The review UIs are closed to crawlers and the git hosts are not: a NoteDb route, checked against the REST corpus

# research(2026-09). scripts/notedb_parity.py fetches every change NoteDb records as submitted in 2024-11 and 2025-01 on AOSP's platform/hardware/interfaces, over git and on every branch, builds them with the same build_from_change as the REST corpus, and compares the two into datasets/results/notedb-parity-aosp.json. The artifact was regenerated on 2026-09-25 from the branch as reviewed: the enumeration, field, example, rebase and timing figures below are from that run, and figures from earlier runs are dated where they appear. The first run's (2026-09-23, main only) are superseded.

FDLoRA's implementation, corrected before it ran: what job 183579 would have measured wrong

An independent review of PR #36 found the round-0 average (Algorithm 1, line 7) was computed per source rather than over every client the plan builds, before job 183579 reached its queue slot. The paper's N is every client the server sees; this repository's sources name separate simulated federations for the attack scripts, and the two are not the same population. Fixed by flattening plan into one list before Stage 1 runs and averaging over all of it, so theta_s0 is one value shared across every client regardless of source. Job 183579 was cancelled as invalid before it produced a result; nothing from it is quoted anywhere.

A rebased test reached chromium-review; the suite is now offline and REST refuses unpermitted hosts

What happened. #35 gave Chromium a REST host. #39's test that an organization without one is not fetched over REST (fetch --org chromium) therefore stopped being refused once #39 was rebased onto #35, and ran a real fetch. Two local make verify runs in #39's worktree each sent at least one request for status:merged after:2025-10-01 before:2025-11-01 to chromium-review.googlesource.com/changes/. How many attempts each made, and what came back, was not recorded and cannot be reconstructed: the retry budget bounds one URL's attempts, not a run's, and the second run held a connection for about eleven minutes before it was killed. No snapshot from the test survives: pytest keeps its recent runs' temporary directories, and none holds one (checked). Writing the guard's own test first then opened one TCP connection to a Google address (no request sent) and made one DNS lookup. The rebased branch was never pushed, so CI sent nothing. This is a breach of the collection stop of 2026-09-22, caused by an unfaked network path in a test.

  • in Corpus
  • 1 min read

Both corpus v2 manifests reproduce from the refined examples

The build and refine rule digests make a month built or refined under other code unreadable, but dedup and split sit under neither, so a change to either would leave the frozen windows intact and verify clean: the failure that let Qt verify at 10,695 while the pipeline yielded 10,692. verify --reproduce reruns dedup and split in memory from the refined examples and compares every window byte for byte with its frozen split file, over every window either side names, and the dedup counts with the manifest. Comparing ids alone would pass a dedup that kept a different copy of an id, and comparing only the rerun's windows would pass one dropped from the bounds (both found in review). On the corpus v2 data, OpenStack and Qt both reproduce byte for byte. The check takes minutes on Qt, so it is a flag rather than the default.

The audit's kappa is the prevalence paradox: AC1 0.853, and the raters part mostly on whether an item is valid

Rater A calls 311 of 383 items valid and rater B 323, so both raters' marginals sit on one label and chance agreement under kappa is high. That depresses kappa at high raw agreement (Feinstein and Cicchetti 1990), which is why kappa is reported beside raw agreement and a paradox-resistant coefficient rather than alone (James, LREC 2026, arXiv 2603.06865). Gwet's AC1 takes chance agreement over the whole five-label scale (Gwet 2008). Same items, same bootstrap seed, so the kappa intervals reproduce byte for byte (label-audit-v2.json, regenerated):

Label audit v2: two blind model raters agree at kappa 0.56, and validity does not differ between the units a contrast compares

The first audit's context-dependent class folded two questions together: whether the request can be understood from what the example shows, and whether the rewrite uses names the example does not show. A clear request answered with a project's own constant is a valid label that no prompt alone reproduces, which is house-style knowledge an adapter can learn, not label noise. Rubric version 2 asks the two separately (scripts/label_audit/rubric.md).

Correction: every successor is a rework, and corpus v2 is refrozen under the corrected rules

Retracted: the rebase-only-successor finding in the two entries above (0.34% of OpenStack's kept examples and 1.65% of Qt's in the audit, 20 and 189 in the v2 table). It was an artifact of the audit's own lookup, which keyed a change's revisions on its Change-Id. A cherry-pick carries the Change-Id to another branch, so a stable-branch copy that was only rebased lent its kind to the original. Resolved per change (by number, else by Change-Id, project and creation time together), every OpenStack successor and 11,444 of Qt's 11,448 are reworks; the other 4 are changes that cannot be told apart and are kept and counted. The rule stays as a guard, and now removes nothing. A review of the v2 branch caught it. Rebase edits carried inside a reworked successor remain the open question the audit entry names.