Research log¶

Contamination battery: what the time partition actually rests on

research(2026-09). The design says the corpus "starts 2024-10-01 against a base model published 2024-09-17" and calls the time partition the weakest of the three methods because it rests on a publication date. It rests on less than that.

PILOT RERUN: the number to quote

Job 143893, same registered model and budget, with every confound the review found removed: deduplicated first (174 of 201 kept), prompt through render_chat for training and both arms, every training example seen exactly twice, LoRA init seeded before attachment. Change-grouped split: 88 changes to 70 train / 18 held out, 147 / 27 examples, 0 of 27 held-out examples repeating a training pair.

PILOT RESULT: the exact-match pass rule survives contact with the corpus

Job 143496. Qwen2.5-Coder-7B, 201 well-posed OpenStack examples, split by change: 91 changes to 72 train / 19 eval, 156 / 45 examples, zero leakage verified.

The NaN was one pair of examples, not the framework

Seven diagnostic jobs, roughly two minutes each. Worth recording because the first conclusion was wrong and the real one is unintuitive.

transformers 5 removed warmup_ratio

TrainingArguments keeps only warmup_steps. The declared budget keeps the ratio, because it is scale-invariant and pinning steps would mean a change in corpus size silently changed the warmup fraction on a pre-registered parameter. training.warmup_steps derives the step count at construction.

The 7B base also floors at exact match 0

The first pilot ran the base evaluation before dying on an API change, which is enough to settle one question:

7B throughput measured on a GH200

The first probe reported 1.0 tok/s and that was an artefact: it generated exactly once, so the entire Triton JIT compilation landed inside the only timed call. Separating warmup from steady state moved the number by a factor of forty. Quoting the first figure would have over-sized every allocation by the same factor.

Grid sizing, at 64 output tokens per example, 14 evaluations, 6 training runs at roughly three times inference cost for two epochs:

The current job_for_grid default of 12 hours therefore covers a test window of roughly 500, which is in range for the measured yield of ~733 examples per organization-month. It is not comfortable, and a longer window needs either a longer wall clock or batching.

Batching is the obvious lever and is not yet used. Peak memory was 15.28 GB on a 102 GB card, so about six copies of the model would fit; batch-of-one is simply what the benchmark measured. A batched evaluation loop should cut the evaluation half substantially. Worth doing before the confirmatory run, not before the pilot.

Structurally ill-posed examples: 27% of the corpus

Independent of the metric question, and actionable now. Inspecting predictions against references showed several examples no model could answer, so the whole corpus was profiled:

Exact match on real hunks: a base-model floor, not a broken metric

First run of the whole measurement path over real examples rather than toy ones. Qwen2.5-Coder-1.5B, 30 real OpenStack hunks, 12 changes, no adapter:

What the pilot may and may not be used for (research(2026-09))

Checked before running a pilot power analysis, and it changes how the result may be reported.

urlopen raises on 5xx, which bypassed the retry budget

http_transport originally let urllib raise. urlopen raises HTTPError on 4xx and 5xx, so a retryable 500 from Qt propagated as a fatal exception and the retry logic in gerrit._get never saw it, despite 500 being in its retryable set. The transport now returns (status, headers, body) for error responses too, which is what makes the budget and Retry-After handling real rather than decorative.

Qt and OpenStack page very differently

Qt caps a page at 10 changes regardless of n. Both months now fetched for real:

Sixteen times the round trips for 1.6x the changes. Both totals match the figures the direction spec quotes, so the queries select what they were meant to.

The pre-cutoff drop rates differ substantially between the two organizations, 29% against 14%. That is worth reporting rather than averaging away: it means the after: operator's update-time semantics bite the two instances differently, presumably because their review cadences differ.

The paging loop is already robust to this because it advances by len(page) rather than by the requested size, so nothing breaks; it is a cost and pacing fact, not a bug. It does mean the two organizations cannot be fetched on the same assumptions, and a Qt fetch wants to be resumable rather than one long run.

Example yield, both organizations

Qt yields fewer examples per change but more in total, because it has twice the changes. Over a ten-month training window that is roughly 7,000 for OpenStack against 11,000 for Qt.

This is a sampling constraint, not just a statistic. Matched training set size is a controlled variable: without it the larger organization's adapter could win for reasons unrelated to conventions. So the usable per-organization corpus is bounded by the smaller, OpenStack, and Qt gets downsampled to match. The number to plan against is therefore OpenStack's, not the total.

Drop profiles differ too. Qt shows no final-patch-set drops in this sample and a higher rate of comments with no line anchor. Worth reporting per organization rather than pooled.

Training budget pinned (research(2026-09))

A Stage 1 pre-registration item rather than a tuning knob: if the budget differed between arms, the comparison would measure the budget rather than the organization.

Two design changes from current literature (research(2026-09))

Hunks now carry surrounding context. arXiv:2607.25851 (Rethinking Training Data for Generating Code Review Comments, read 2026-09-14) names three sources of misalignment between a code change and its review comment: semantic ambiguity, lack of actionability, and context dependence. The third is a real gap here, and it is the same root cause as a failure already measured: a bare hunk gave the model no way to know its indentation level, so a rewrite that was otherwise exactly right lost strict exact match. Context is prompt material only; the target stays the hunk and scoring compares only the hunk.