The permutation test was comparing two different statistics¶
Correcting the entry above. The project-level permutation read its observed value from the detector run made earlier at eight reference splits and 600 draws, while every relabeling was scored at two splits and 150. The true grouping is one of the 84 arrangements, so under a matched comparison it scores against itself and the p-value cannot fall below 1/84; the mismatch let it score below its own recomputation, and at 128 examples per client the test printed p = 0.000, which an enumeration of 84 arrangements cannot produce. That impossible value is what exposed it.
Both sides now go through one call, and the enumeration is checked to contain the true grouping before a p-value is reported at all.
| within C++, project-level permutation | as recorded | corrected |
|---|---|---|
| AOSP, 64 examples per client | 0.012 | 0.012 |
| Qt, 64 examples per client | 0.060 | 0.238 |
AOSP's result stands. Qt's does not: the 0.060 that read as near-significant was an artefact of the two settings, and Qt's true grouping is unremarkable among relabelings of the same nine projects at this training length. The claim that survives is the narrower one -- one of the two organizations is detectable beyond its projects, not both.