Running on a Cluster¶

While Pharos core generation and gating run on local workstations, cluster execution (e.g. Slurm HPC environments) is required for two workloads:

1. Large Models (>8B Parameters): e.g. 14B at Q4 (~9 GB VRAM) exceeds 8 GB consumer GPUs.
2. Adapter Fine-Tuning: LoRA fine-tuning for 7B models requires 16–24 GB VRAM.

Execution Environment Breakdown¶

Model-dependent numbers are not bit-reproducible across different CPU/GPU platforms (e.g., small numerics variations under temperature 0). The platform is part of the measurement provenance.

Slurm Cluster Pipeline¶

# 1. One-time environment setup
bash cluster/setup-env.sh

# 2. Verify non-GPU codebase (CPU-only Slurm node)
sbatch cluster/verify.sbatch

# 3. Verify PyTorch & PEFT adapter setup on GPU
sbatch cluster/gpu-probe.sbatch

# 4. Prefetch model weights (CPU job) and chain GPU evaluation job
PRE=$(sbatch --parsable cluster/prefetch-models.sbatch)
sbatch --dependency=afterok:$PRE cluster/sweep.sbatch

High-Impact Slurm Traps¶

⚠️ Trap 1: Thread Oversubscription¶

Slurm cgroups restrict --cpus-per-task (e.g., 8 cores allocated on a 96-core node). By default, numpy and scikit-learn inspect total system cores (96), resulting in \(12\times\) thread oversubscription and severe performance degradation.

{"event": "run.context", "machine_cpus": 96, "usable_cpus": 8, "thread_limits": {"OMP_NUM_THREADS": null}, "oversubscription_risk": true}

⚠️ Trap 2: Session-Bound Interactive Jobs¶

Interactive srun sessions or backgrounded ollama pull commands terminate if the parent SSH connection times out.

Verified Cluster Achievements¶

• ⚠️ Cross-machine gate reproducibility: repeated 2026-08-04 on the corrected corpus, and the earlier bit-for-bit claim did not survive. The corpus is bit-identical -- its SHA-256 agrees at every seed -- but 2 of 7 surface baselines differ, by 1.2e-05 and 6.5e-06, with numpy, scikit-learn, scipy and the BLAS at identical versions. The BLAS selects kernels by processor (AVX-512 on the Xeon, AVX2 on the Ryzen), changing reduction order inside the probe fit. No verdict moves: the gap to the 0.720 ceiling is ~4 orders of magnitude larger. Repeat it with sbatch cluster/gate-determinism.sbatch and scripts/measure_gate_determinism.py --compare.
• ✅ Gradient Learnability: Validated on NVIDIA A100-PCIE-40GB (torch 2.13.0+cu130, peft 0.20.0). LoRA adapter fine-tuning moved triage F1 from 0.469 to 1.000.

Hugging Face Credentials & Caching¶