The sealed window is about twice the size the power analysis assumed¶
The power analysis took 880 OpenStack changes and 2,400 Qt for the test window, scaled from the dev window, which the capture model says is the worst available base. The train window, missing 5.0% and 4.2%, is the right one, which frees the dev window to be a held-out check on the model rather than an input to it.
| train changes | implied true rate | dev predicted | dev actual | miss | |
|---|---|---|---|---|---|
| OpenStack | 1,737 over 10 months, captured 0.950 | 182.9/month | 223 | 235 | -5.3% |
| Qt | 3,160 over 10 months, captured 0.958 | 329.9/month | 195 | 175 | +11.6% |
Both organizations now pass, and Qt's passing is what tells the story. The first version of this analysis predicted Qt's dev window at 527 against an actual 175 and printed the 201% miss as informational text, then used the same model on the next line to project the sealed window. The miss was not Qt being strange: it was the two input bugs in the capture model, a change key that admitted cherry-pick siblings and a horizon one month too long. With those fixed the same check lands at +11.6% with nothing else changed. A check that can only be read is not a check, so it now refuses to project the sealed window when the miss exceeds 15%.
| test window projection | changes | power analysis assumed | ratio |
|---|---|---|---|
| OpenStack | ~1,809 | 880 | 2.06x |
| Qt | ~3,281 | 2,400 | 1.37x |
So the registered MDEs are conservative rather than optimistic. Two things the check cannot separate: it tests the capture model and a constant arrival rate together, and the arrival rate is visibly not constant (OpenStack's example-bearing changes per created month range 136 to 232). A pass is therefore evidence that the two errors are small or cancel, not that either is zero.