Repack the same clients and the detector survives; the classifier does not¶
*-c128-p2.json. The 128-example run said leakage is a function of how long each client trains.
It was one packing, so who shared a client with whom was decided once, and the reading rested on
two points that differ in size as well as in length. This is the same 51 clients, the same
training seed, the same 128 examples each, repacked with packing seed 2: the only thing that moves
is which examples group into which client.
| within C++ | 64 | 128, packing 1 | 128, packing 2 |
|---|---|---|---|
| organization, nearest class | 0.395 (majority 0.698) | 0.765 (majority 0.735), p 0.034 | 0.559, p 0.455 |
| organization, beyond project | 0.395, p 0.75 | 0.676, p 0.19 | 0.529, p 0.55 |
| project, all clients | 0.442 | 0.373 | 0.412 |
| content, all clients | 0.909 | 0.784 | 0.961 |
| AOSP detector, rounds of 16, two clients | AUC 0.858, TPR 0.364 | AUC 0.940, TPR 0.588 | AUC 0.887, TPR 0.419 |
| Qt detector, rounds of 8, two clients | AUC 0.859, TPR 0.456 | AUC 0.980, TPR 0.910 | AUC 0.962, TPR 0.813 |
| AOSP project permutation, median of 4 seeds | 0.012 | 0.012 | 0.012 |
| Qt project permutation, median of 4 seeds | 0.250 | 0.018 | 0.042 |
The length effect replicates where it was claimed, and only there. Every detector cell at 128 beats its 64 counterpart under both packings: AOSP 0.364 to 0.588 and 0.419, Qt 0.456 to 0.910 and 0.813. AOSP's permutation sits at the floor in all twelve seed-by-packing cells, and Qt's, which could not be told from a relabeling at 64, is below 0.05 at the median under both packings.
The classifier rows were a packing artefact. Nearest-class attribution within C++ went 0.395 to 0.765 and read as the classifier waking up; repacked it is 0.559, below its own 0.735 majority, at p 0.455. The beyond-project control moves with it, 0.676 to 0.529. Nothing about the organizations or the training changed between those two numbers, so the difference is which examples happened to share a client, and the earlier caveat that 0.765 was one client's difference on 34 was the right instinct: the resolution was never there.
What this settles for RQ2. The registrable claim is the detector's, not the classifier's: at one false alarm in a hundred, a round holding two Qt clients is caught 81 to 91 percent of the time when clients train on 128 examples against 46 percent at 64, and the ordering holds under a repacking that halves the classifier's accuracy. That is the same operating-point distinction the rest of RQ2 rests on, arriving here from a control rather than from an argument.
The third length, which asks whether the detector keeps climbing, is still the open item. It now needs two packings to be worth running.