Apparatus¶

The 7B base also floors at exact match 0

The first pilot ran the base evaluation before dying on an API change, which is enough to settle one question:

7B throughput measured on a GH200

The first probe reported 1.0 tok/s and that was an artefact: it generated exactly once, so the entire Triton JIT compilation landed inside the only timed call. Separating warmup from steady state moved the number by a factor of forty. Quoting the first figure would have over-sized every allocation by the same factor.

Grid sizing, at 64 output tokens per example, 14 evaluations, 6 training runs at roughly three times inference cost for two epochs:

The current job_for_grid default of 12 hours therefore covers a test window of roughly 500, which is in range for the measured yield of ~733 examples per organization-month. It is not comfortable, and a longer window needs either a longer wall clock or batching.

Batching is the obvious lever and is not yet used. Peak memory was 15.28 GB on a 102 GB card, so about six copies of the model would fit; batch-of-one is simply what the benchmark measured. A batched evaluation loop should cut the evaluation half substantially. Worth doing before the confirmatory run, not before the pilot.

Exact match on real hunks: a base-model floor, not a broken metric

First run of the whole measurement path over real examples rather than toy ones. Qwen2.5-Coder-1.5B, 30 real OpenStack hunks, 12 changes, no adapter:

What the pilot may and may not be used for (research(2026-09))

Checked before running a pilot power analysis, and it changes how the result may be reported.

urlopen raises on 5xx, which bypassed the retry budget

http_transport originally let urllib raise. urlopen raises HTTPError on 4xx and 5xx, so a retryable 500 from Qt propagated as a fatal exception and the retry logic in gerrit._get never saw it, despite 500 being in its retryable set. The transport now returns (status, headers, body) for error responses too, which is what makes the budget and Retry-After handling real rather than decorative.

Qt and OpenStack page very differently

Qt caps a page at 10 changes regardless of n. Both months now fetched for real:

Sixteen times the round trips for 1.6x the changes. Both totals match the figures the direction spec quotes, so the queries select what they were meant to.

The pre-cutoff drop rates differ substantially between the two organizations, 29% against 14%. That is worth reporting rather than averaging away: it means the after: operator's update-time semantics bite the two instances differently, presumably because their review cadences differ.

The paging loop is already robust to this because it advances by len(page) rather than by the requested size, so nothing breaks; it is a cost and pacing fact, not a bug. It does mean the two organizations cannot be fetched on the same assumptions, and a Qt fetch wants to be resumable rather than one long run.

Example yield, both organizations

Qt yields fewer examples per change but more in total, because it has twice the changes. Over a ten-month training window that is roughly 7,000 for OpenStack against 11,000 for Qt.

This is a sampling constraint, not just a statistic. Matched training set size is a controlled variable: without it the larger organization's adapter could win for reasons unrelated to conventions. So the usable per-organization corpus is bounded by the smaller, OpenStack, and Qt gets downsampled to match. The number to plan against is therefore OpenStack's, not the total.

Drop profiles differ too. Qt shows no final-patch-set drops in this sample and a higher rate of comments with no line anchor. Worth reporting per organization rather than pooled.

Training budget pinned (research(2026-09))

A Stage 1 pre-registration item rather than a tuning knob: if the budget differed between arms, the comparison would measure the budget rather than the organization.

Two design changes from current literature (research(2026-09))

Hunks now carry surrounding context. arXiv:2607.25851 (Rethinking Training Data for Generating Code Review Comments, read 2026-09-14) names three sources of misalignment between a code change and its review comment: semantic ambiguity, lack of actionability, and context dependence. The third is a real gap here, and it is the same root cause as a failure already measured: a bare hunk gave the model no way to know its indentation level, so a rewrite that was otherwise exactly right lost strict exact match. Context is prompt material only; the target stays the hunk and scoring compares only the hunk.

First honest example count

build run against the real October 2024 OpenStack snapshot, 476 of 1,678 changes processed in 95 seconds before a time cap.

Half the "review comments" are not reviews

Measured on live OpenStack data, 62 code-file comments:

Next: plan A2, stage bodies

Wire fetch, build, dedup, split and freeze to real artifacts under datasets/gerrit/<org>/.

Repo split from phalanx-fl

The corpus and measurement code was built inside phalanx-fl and moved here. It had grown to 80% of that repo's test suite while importing exactly one function from it, in a repo whose stated purpose is a Flower testbed.