Skip to content

The interval's own false-positive rate, measured rather than quoted

datasets/results/interval-calibration.json, scripts/interval_calibration.py. Section 7 of the report states the pairs cluster bootstrap's false-positive rate under a true null. That figure has been living in a comment in measure/stats.py since it was measured, with no artifact behind it, which is the one thing a registered report cannot do with a number a reviewer will check.

It is now measured by the script, against the same cluster_bootstrap the gate calls, with the dev window's own change sizes and the matched adapter arm's accuracy: 1,500 trials a point, 2,000 resamples, two-sided exclusion of zero against a nominal 5%.

The rate the null is drawn at decides the answer, and the first run used the wrong one. At the base arm's 0.048 the interval excludes zero 2.9% of the time at 19 clusters, which reads as comfortably conservative. The gate does not contrast base arms; it contrasts two adapter arms near 0.32, where the same interval returns 5.7%. A binary outcome's variance collapses at the extremes, so calibration measured at an accuracy the study never operates at answers a question nobody asked. The script now takes its rate from the matched adapter arm and sweeps the dependence explicitly.