The first run using the train and dev windows rather than a random holdout from one month.
Both organizations, base plus both adapters on both dev windows, equalized training, one
seed. datasets/results/rq1-windows.json. Completed in 2:19:50 against a 4:30 request; all
seven outcome-neutral checks pass and the apparatus holds.
The power analysis took 880 OpenStack changes and 2,400 Qt for the test window, scaled from the dev window, which the capture model says is the worst available base. The train window, missing 5.0% and 4.2%, is the right one, which frees the dev window to be a held-out check on the model rather than an input to it.
datasets/results/window-report-{openstack,qt}.json.
Thirteen months, 2024-10 to 2025-10, collected and built in 3h32m.
datasets/results/window-report-openstack.json.
Building the train and dev windows stalled twice against review.opendev.org. DNS was healthy throughout (16 timed lookups, none failed). Failed requests completed the lookup and never completed the TCP handshake, including with the address pinned. After hours of unpaced building, at about 20 requests a second, a quarter to a third of new connections were dropped. After 20 minutes of sending nothing, 0 of 30 were. Qt's server showed no such pattern.
Snapshots select merged changes by last update (Gerrit after:/before:), windows assign
them by creation, and the seal refuses every month from 2025-11 on. So a change created in
a collected window whose last update falls on or after 2025-11-01 is in no snapshot. The loss
is concentrated in the changes reviewed longest, near the boundary.
scripts/power_rq1.py datasets/results/rq1-pilot-equalized.json --size openstack=880 --size qt=2400,
on the equalized RQ1 pilot's per-change outcomes.
Job 143496. Qwen2.5-Coder-7B, 201 well-posed OpenStack examples, split by change: 91 changes to 72 train / 19 eval, 156 / 45 examples, zero leakage verified.
Independent of the metric question, and actionable now. Inspecting predictions against references showed several examples no model could answer, so the whole corpus was profiled:
o=ALL_FILES returns only metadata
(lines_deleted, old_sha, size_delta). Hunks now come from Gerrit's own diff
endpoint, which also keeps hunk boundaries identical to what the reviewer saw.after: filters on last update, not creation. after:2024-10-01 before:2024-10-03
returned a change created 2024-08-26. The post-cutoff contamination argument rests on
creation date, so created_on_or_after enforces it client-side.| changes returned by the query | 2,350 in 24 pages, 13s |
| dropped as created before the cutoff | 672 (29%) |
| kept | 1,678 |
| snapshot size | 503 KB gzipped NDJSON |
| owners reduced to a pseudonym only | 1,678 / 1,678 |
The 2,350 matches the figure the direction spec quotes for October 2024, so the query is
selecting what it intended. The 29% is the after: bug measured rather than argued:
without the client-side created_on_or_after filter, 672 changes created before the
base model's release date would have entered the corpus, and the contamination argument
would have been false while appearing to hold.
The first real fetch. Numbers the Stage 1 report's sampling section needs, and which no amount of reading the API docs would have produced.