Skip to content

Planned before any run: how much of H1 is the partition

The registered split is sensitive to small corpus changes (117 of 238 projects moved between v2 and v3), so H1 reads one partition of many that meet the same criteria, and nothing measured so far says how much the reading depends on which. Fixed here, before any job:

  • Partitions. assign(..., order_seed=s) shuffles the project names with seed s and gives each to the smaller half, as the registered rule does largest-first. Seeds are taken in order from 1, skipping any whose halves fail split_criteria against OpenStack's registered halves (size floor 2,004, ceiling 0.3187, the project and share rules), until four qualify. The registered split is the fifth.
  • Runs. Each alternative partition at training seeds 1 and 2, on corpus-v3/openstack, the placebo gate unchanged; the registered partition's seeds 1 and 2 are the v3 runs already made.
  • Reading. Per run, H1's point estimate (the two halves' own-minus-sibling contrasts, equally weighted). A two-way layout, five partitions by two seeds: the between-partition variance of the partition means, less the within-partition (seed) variance over two, is the partition component, set beside the seed effect. Reported with the partitions' spread, not tested.
  • What it decides. If the partition component is comparable to the seed effect or larger, H1 as registered is conditional on an arbitrary partition, and the Stage 1 design moves to repeated splitting (several partitions, the median estimate and median bounds at an adjusted level, after Chernozhukov, Demirer, Duflo and Fernández-Val, Econometrica 2025); if it is small, one partition stands and the variance is stated.

Selection, applied as stated: seeds 1 to 30 were checked; 2, 8, 13 and 29 qualify. Of the other 26, 22 fail only the size floor (a shuffled order balances less tightly than largest-first, and the floor is the registered split's smaller half), 3 fail both the floor and the language-mix ceiling (seeds 3, 7, 21), and 1 fails the ceiling alone (seed 22). Jobs 207730 to 207735, 207739 and 207740, pinned at a41af73.