The seal right-censors the collected windows: dev loses about a fifth¶
Snapshots select merged changes by last update (Gerrit after:/before:), windows assign
them by creation, and the seal refuses every month from 2025-11 on. So a change created in
a collected window whose last update falls on or after 2025-11-01 is in no snapshot. The loss
is concentrated in the changes reviewed longest, near the boundary.
Estimated from OpenStack's collected snapshots (25,191 merged changes, 2024-10 to 2025-10). Creation-to-last-update lag, from changes created 2024-10 to 2024-12 (ten or more months of follow-up, so lags under nine months are fully observed): median 3.8 days, p90 71.8, p99 272.3; P(lag > 7 d) 0.391, > 14 d 0.291, > 30 d 0.196, > 61 d 0.116, > 90 d 0.077, > 180 d 0.027. Expected share of each creation window's true cohort missing:
| window | collected changes created there | estimated share missing |
|---|---|---|
| dev, 2025-09 to 10 | 4,104 | 21.6% |
| train, 2025-08 | 1,820 | 9.5% |
| train, 2025-06 to 07 | 3,452 | 5.4% |
| train, 2024-11 to 2025-05 | 13,452 | 1.4% |
Approximations: creation days are taken from the collected, already truncated changes, and last update includes post-merge comments; both probably understate the loss.
Qt, same method (52,895 merged changes; early cohort 11,438): lag median 0.4 days, p90 15.5, p99 128.0; P(lag > 7 d) 0.172, > 30 d 0.058, > 90 d 0.018, > 180 d 0.006.
| share of each window's true cohort missing | OpenStack | Qt |
|---|---|---|
| dev, 2025-09 to 10 | 21.6% | 7.9% |
| train, 2025-08 | 9.5% | 2.2% |
| train, 2025-06 to 07 | 5.4% | 1.1% |
| train, 2024-11 to 2025-05 | 1.4% | 0.2% |
The censoring is unequal across organizations. Qt reviews close far faster, so the seal removes almost three times as much of OpenStack's dev window. RQ1 compares organizations, so under creation-based windows this is a confound rather than shared noise: the two dev windows are drawn from differently truncated populations. It strengthens the case for assigning windows by last-update month, where neither organization is censored.
Why it matters. The missing changes are the slow, long-reviewed ones, which plausibly carry different review comments. The dev window, and the late train months, are biased toward quick reviews.
Options, a Stage 1 decision (not changed in code):
- Assign windows by last-update month rather than creation. Collection and assignment align and nothing is censored; changes stay disjoint, but review activity near a boundary overlaps.
- A gap between dev and test that excludes changes created in the final months before the seal, the standard embargo for temporal splits; shortens a window and so power.
- Keep creation windows, report the bias, and register a follow-up horizon for the test window: after acceptance, fetch through its end plus about six months (under 3% missing).