What the design can detect: a sensitivity analysis, replacing power at an observed effect¶
Superseded. These sizes and bounds belong to the retired organization-against-organization gate on corpus v1; the current bounds are the fixed-K simulation's (2026-09-30), and its test size is re-projected over twelve months (audit, 2026-09-30).
datasets/results/sensitivity-b0.0098.json, scripts/sensitivity.py. The test window's size is
fixed by what exists, so the Stage 1 question is not "what is the power" but "what is the smallest
effect this design detects". Power computed from a pilot's own estimate is biased upward, the more
so when the pilot looked promising (Albers and Lakens, JESP 2018; Lakens, Collabra 2022), which is
why the 0.833 figure is withdrawn rather than updated.
Simulated at the projected test-window sizes, under the crossed interval, at the seed main effect's one-sided 95% upper bound of 0.0098, with marginal power 0.894 per organization so the conjunctive gate reaches about 0.80:
| organization | changes | three seeds | five seeds |
|---|---|---|---|
| OpenStack | 2,171 | +0.0235 | +0.0210 |
| Qt | 3,937 | +0.0211 | +0.0179 |
The design detects about two exact-match points per organization. Two more seeds buy 0.002 to 0.003, which is why three is registered: the seed term is already small beside the change term at these window sizes.
Bracketed by the seed effect's own uncertainty (sensitivity-b0.json), since three seeds
estimate a variance poorly and the honest statement is a range rather than a figure:
| seed main effect | OpenStack | Qt |
|---|---|---|
| 0.000, the point estimate | +0.0160 | +0.0129 |
| 0.0098, the 95% upper bound | +0.0235 | +0.0211 |
So the design resolves somewhere between 1.3 and 2.4 exact-match points. Qt's dev-window effect (+0.031) is above the whole range and OpenStack's (+0.017) inside it, which is why one organization clears zero and the other does not, and why the Stage 1 report should state the range.
Read against the dev window, this predicts the outcome we have. Qt's dev-window effect is +0.031, above its threshold; OpenStack's is +0.017, below its own. The gate came out mixed on exactly that pattern, and a test window that behaves like the dev window would return mixed again. That is a statement about what the study can resolve, and it is better made now, in the Stage 1 report, than discovered afterwards.