Skip to content

The checkpoint's repetition penalty, measured: plain greedy is registered

# research(2026-10). scripts/decoder_read.py, datasets/results/decoder-penalty.json. The three rescored partition runs ran at 3264e08, the pre-rebase copy of 09e1511, through the first --reuse-adapters, which records no test 3. Qwen2.5-Coder-7B-Instruct's pinned generation_config.json sets repetition_penalty: 1.1, and transformers 5.17.0 adds its processor whether or not it samples, over the prompt's tokens as well as the output's. Every result before this entry therefore decoded greedy at 1.1, while the registration says greedy. A code refinement is mostly a copy of the code under review, and a penalty on prompt tokens penalises the copy. Measured on stored adapters, rescored with only the decoder changed (decoder_check.sbatch, decoding_check.sbatch, fp32):

  • The control reproduces. OpenStack partition 2 rescored at 1.1 in a new job keeps every prediction of the stored run in every arm, and the calibration adapter rerun greedy at 1.1 keeps all 449 and 488. Rescoring isolates the decoder.
  • Without the penalty, exact match rises, most for the base model. On partition 2 the base model goes from 0.031 to 0.078 on half a and 0.053 to 0.061 on half b; on partition 3 from 0.036 to 0.059 and 0.045 to 0.075. Adapter arms move by -0.006 to +0.036. The base arms change 59 to 63% of their predictions, the adapter arms 37 to 47%.
  • Per-run contrasts move, partition 3's by more than the SESOI. Partition 2's own-minus-sibling on half a goes from 0.000 to -0.008 and half b stays at +0.041, its interval now above zero; partition 3's half a goes from +0.024 to +0.041 (interval above zero) and half b from +0.018 to 0.000. Both runs' pilot verdicts move from fail to mixed.
  • The planted calibration reverses. With the annotation on 25.1% of training refinements, the adapter emitted it on 0.628 and 0.650 of greedy outputs at 1.1. Sampled at temperature 1 without the penalty, the adapter's unaltered distribution, it emits 0.243 and 0.219, near the trained rate; at 1.1, 0.501 and 0.467. Greedy without the penalty emits 0.029 and 0.020. The penalty made the overshoot, and plain greedy decoding, being mode-seeking, drops a minority convention.

Decided by standard: plain greedy, penalty 1.0, passed explicitly (model.REPETITION_PENALTY). It is what the registration describes and the neutral value, the 1.1 came with a checkpoint tuned for chat sampling rather than chosen, and it distorts exactly what the study measures, how much of a convention an adapter reproduces. Stage 1 is not submitted, so the registered decoder can still change; the cost is inference only, since adapters are reused. A result without a recorded penalty reads as 1.1 (runs.UNRECORDED_REPETITION_PENALTY), and runs decoded differently are refused together by runs.merge and partition_pilot.py.

Rescored at 1.0 before anything is read again: every development partition run and its planted variants on both organizations, which re-derives K, the sensitivities and the pilot figures; the calibration ladder's emission readings; the retrieval and rules comparators, whose cache key now carries the penalty; and the contamination battery's guided completion. The determinism measurement behind the fp32 registration is repeated at 1.0. Plain greedy's suppression of a minority convention bears on what the calibration ladder can show, and is read when it is rerun.