Inference numerics registered: fp32¶
datasets/results/determinism-sym-0-gh-a-081.json, -gh-a-003.json. The base model over the same
150 prompts, four passes a job, two jobs on different nodes.
| comparison | predictions differing, of 150 |
|---|---|
| fp32, the two jobs | 0 |
| bf16, the two jobs | 5 to 7 |
| bf16, repeated inside one job | 0 to 2 |
| bf16, a reloaded model inside one job | 1 to 2 |
| bf16 against fp32 | 17 to 19 |
bf16 greedy decoding does not reproduce, even within one process, which is what the two nulls' disagreement was. fp32 reproduces exactly across jobs and nodes, matching Yuan et al. (arXiv:2506.09501v2, read from the paper): "FP32 precision consistently achieves near-perfect reproducibility with negligible variance, FP16 shows moderate variability, while BF16 exhibits substantial instability", and their recommendation, "If using greedy decoding with a single run, please use FP32 precision to improve the reproducibility of your results". Their LayerCast keeps weights in bf16 and computes in fp32 to save memory; this upcasts the weights as well, which a GH200 has room for at 7B, and is the stronger of the two. The earlier reading that the node was the variable is wrong: a fresh job on the stored run's own node differs from it as much as a job on another node.
HFGenerator now upcasts the bf16 weights to fp32, exactly, before generating, and every result
records inference_dtype. This is the rule fixed in the ROADMAP before the jobs ran: fp32 if bf16
differs between nodes and fp32 does not.