The contamination battery separates the windows, on fifteen times the sample¶

datasets/results/contamination-openstack-6mo-with_context.partial.json. Six post-cutoff months (2024-10 to 2025-03) against the six-month control (2023-08 to 2024-01), built by one pipeline under one cutoff, scored on hunks with context on the base Qwen/Qwen2.5-Coder-7B. The job was still running guided completion when these were read: the membership scores are written before the generative half starts, which is the change made after job 145093 lost exactly this result to its wall.

1,971 post examples over 797 changes, 2,503 pre over 877. The point gap is compare_windows, the battery's own definition; the interval resamples changes within each window independently, since the windows are separate populations rather than paired.

The first interval in this battery's history to exclude zero, and it points the right way. A higher Min-K%++ score means the model finds the text more familiar. The pre-cutoff control, which predates the checkpoint's release and plausibly sits in its training data, reads as more familiar than six months the model cannot have seen. That is the direction membership predicts for a known-member control.

The point estimate held still between the last two runs while the interval tightened around it until zero fell outside. That is what more sample does to an effect that is there, and what it does not do to noise, which on the first run had the sign backwards.

What it supports: less than the first reading of it claimed. This entry originally called the gap evidence that the instrument works. A control run an hour later withdrew that.

Meeus et al. (SoK, SaTML 2025, arXiv:2406.17975) show that membership benchmarks built post hoc, with members and non-members from different periods, carry distribution shifts strong enough that a model-less bag-of-words classifier separates them, and that this invalidates reading the separation as memorization. This battery is such a split, so their control was run on exactly the 4,473 examples Min-K%++ scored, using the separability probe's classifier with changes held out whole:

The blind baseline does at least as well as the membership score, and probably better. Balanced accuracy is a lower bound on a classifier's AUC, so the comparison is tilted toward Min-K%++, and it still does not come out ahead. By the SoK's criterion the -0.074 gap cannot be told apart from ordinary drift across eighteen months of OpenStack: the windows differ in what they are about, and that alone is enough to produce a gap this size.

So the battery neither shows exposure nor validates the instrument. It is uninformative, which is a legitimate result for a check registered as corroborative, and it means the study's protection against contamination rests where it always did: the post-cutoff windows postdate the checkpoint's release, so they cannot have been in its pretraining data, whatever a membership score says. The SoK's recommended alternatives, randomized splits and injected canaries, need control over the training data, which a released checkpoint does not give.

Min-K% does not separate the windows. Its interval covers zero on the same examples. That is the pattern expected if Min-K%++ is the more sensitive of the two, which is the reason it is the registered primary.

The completed job agrees with the partial, and guided completion stays at its floor. The full contamination-openstack-6mo-with_context.json reproduces both membership gaps to four places, so writing membership first lost nothing. Guided completion reproduced 1.13% of 798 post-cutoff hunks verbatim against 0.88% of 1,027 control hunks, a gap of +0.0025: the floor, as registered. Its edit similarity leans the same way as Min-K%++, 0.364 post against 0.403 pre, and the blind-baseline result applies to it exactly as it does to the membership score, so it is not read as exposure either.