How much access does an agent need to fix a broken ML workload? BrokenML runs nine models against 46 broken jobs at three access levels: configuration files and code only; live system logs; and full root access.
The setup
Each task models a real broken distributed ML job with PyTorch over Gloo (CPU) on a two-node Slurm cluster or a llama.cpp server running as Docker Compose containers. The faults were designed to reflect what you see in practice, including shape mismatches across ranks or hanging collectives, a straggler node, OOM errors, broken model checkpoints, scheduling issues, poor serving throughput, and silent loss divergence.
The agent fixes the job by editing its configuration or code. A separate verifier then spins up a fresh cluster and re-runs the job to evaluate the fix.
Every agent ran every task once in each of three regimes:
- Files only: configuration files and code with a Python sandbox.
- Files + live logs: adds read-only metrics, logs, and a bug report.
- Live cluster: all the above plus root on the cluster and the ability to re-run the job.
Models ran through the Harbor + Terminus 2 harness with high reasoning effort and a 25-minute limit per task on a c7a.8xlarge AWS instance with 32 vCPUs and 64 GB of RAM.
Results
Each model attempted each task only once, so the final ranking should be interpreted cautiously. Tasks were developed with Fable 5.1 and Opus 5.5 coding agents, which may have introduced bias, and pre-tested with Sonnet 5.5 and Qwen 3.8 27B.
For every agent, higher access levels improved performance and made fixing jobs up to 8x cheaper. A cheaper model with higher access can outperform a more capable one; conversely, only the strongest models solved the majority of tasks when access was limited to static configuration files and code. Live logs nearly replace full cluster access for the strongest models.
GPT-6 Luna seems to offer the best price-performance balance for this benchmark, at ~5% of Sonnet 5.5’s recorded price per run.
Some of the tasks require access to the node’s state for correct diagnosis. Other faults, such as incomplete checkpoints, can often be diagnosed from logs. And when tasks are defined exclusively by configuration (serving engine), having access to the files might be sufficient.
What went wrong
- Sonnet 5.5 spent 39% of the bill on thinking that the harness discarded due to an issue in Harbor, so the real per-run Sonnet 5.5 costs are 1.7 times what the harness recorded.
- Haiku 5.5 was released during the benchmarking run and was added to the study; however, it kept getting stuck in a loop, burning through tokens, and was removed.
- Qwen 3.8 27B achieved ~87% in trial runs during testing, exposing run-to-run variability.
- gpt-oss-120b was so expensive and performed so poorly that it was stopped after 32 trials.
- Inference providers’ throughput contributed to agents’ performance and might have affected open-weights models’ results.
- I had to patch Slurm to fix slurmd behaviour in unprivileged containers.
- From interesting leaks found by red-team review and fixed before the evaluation: some Docker hostnames, networks and mounts were derived from task directory names, so agents used them to infer the issue.
Limitations
The benchmark runs on CPU with Gloo; key symptoms were cross-checked on 2×H100 with NCCL, but exposure to real GPU-cluster faults is limited. Pass@1 has high variability; harness, effort level, and quantisation were pre-selected. The live logs regime also received bug report information, which might not be available in a real scenario. The benchmark itself, while providing expected separation between regimes, is saturated for frontier models with full cluster access.
Code is available at zhebrak/brokenml.