Skip to content

Research log

Both halves carrying a convention: the gate passes on both

datasets/results/calibration-sym-0.25.json. The symmetric calibration: half a carries # approved on 25% of refinements, half b carries # reviewed on 25%, equal-length markers, neither half convention-free. This is the realistic case, two organizations each with their own habit, and it is the case the asymmetric sweep could not speak to.

The interval measures one source of variance of two, and a null crossed zero

Corrected later on 2026-09-18. The inference below overreaches. The run-to-run swing is what seed-by-change noise produces (z = -1.47 and 0.00 against it), and a single-seed change bootstrap already carries that noise. The only variance it omits is a seed main effect, which two runs give no evidence of. The retraction of single-seed results and of the conjunctive power is withdrawn pending a measurement. See "The run-to-run swing is noise the interval already carries" below.

Greedy decoding adds to the amplification; it does not cause it

Corrected on 2026-10-08. Neither arm decoded the adapter's unaltered distribution. scripts/decoding_check.py wrote datasets/results/decoding-marker-0.25-t1.0.json under transformers 5.17.0, which applies the checkpoint's repetition_penalty: 1.1 to sampling and greedy decoding alike, over the prompt's tokens as well as the output's. Both rates below are therefore at penalty 1.1, and the claim about the unaltered distribution is withdrawn. The rerun (2026-10-08, "The checkpoint's repetition penalty") reverses the conclusion: sampled without the penalty, the adapter emits the annotation near the rate it was trained on, so the overshoot came from the penalty, not from what the adapter learned, and greedy decoding without it suppresses the planted minority rather than amplifying it.

  • in Corpus
  • 4 min read

The contamination battery separates the windows, on fifteen times the sample

datasets/results/contamination-openstack-6mo-with_context.partial.json. Six post-cutoff months (2024-10 to 2025-03) against the six-month control (2023-08 to 2024-01), built by one pipeline under one cutoff, scored on hunks with context on the base Qwen/Qwen2.5-Coder-7B. The job was still running guided completion when these were read: the membership scores are written before the generative half starts, which is the change made after job 145093 lost exactly this result to its wall.

SPORC as the interim target

Research Computing retired rc-onboard for research jobs on 2026-09-16, and fl-mlm has no TIGRIS association: in ColdFront both fl-mlm and prdiscourse carry Needs Review, and the RC documentation grants every project without a pending review a TIGRIS Slurm account. sacctmgr show assoc lists fl-mlm on sporc only, and sbatch --test-only against the tigris partition fails with Invalid account or account/partition combination.

The contrast is not blind

datasets/results/calibration-marker-1.json. One organization's train window split into two halves that no change spans, a fixed annotation appended to every refinement in the second, and the identical matched-versus-mismatched contrast RQ1 uses.

The cohort-trend worry, tested properly

scripts/quasi_independence.py, datasets/results/quasi-independence.json. Lynden-Bell needs the lag and its truncation limit to be quasi-independent, and the limit here is a deterministic function of creation month, so the assumption is that review latency is stationary over the eighteen months the corpus spans. The log recorded that it fails, citing OpenStack's P(lag<=1) rising from 0.667 in the oldest cohort to 0.905 in a recent one. That citation was wrong, and the conclusion drawn from it was too strong.