Newborn 
PY A benchmark and open call for valuable, reproducible RL runs.
https://thinking-space.github.io/AutoRLBench/ ↗ HEALTH ? 70 / Stable
Newborn PY // readme
A benchmark and open call for valuable, reproducible RL runs. Each task is a real training run that went wrong. The agent has to find the root cause (issue) and, where a repaired run exists, fix it and rerun (solution).
🚀 News
- 2 Nov 2026 — We release the first version of the task list: 18 tasks, 13 of which support solution-level evaluation.
- 24 Oct 2026 — Call for contributions officially open, and Kimi-K3.1-Pro added to the leaderboard.
- 19 Oct 2026 — GitHub repo and website released.
Leaderboard
AutoRLBench Leaderboard, issue + solution evaluation.
| # | Model | Agent | Issue Score | Solution Score | Overall |
|---|---|---|---|---|---|
| 1 | Claude-Fable-5.1 | Claude Code | 71.4 % | 80.0 % | 75.7 % |
| 2 | Gemini-4-Argon | Claude Code | 52.4 % | 80.0 % | 66.2 % |
| 3 | GPT-6.1-Sol | Codex | 61.9 % | 70.0 % | 66.0 % |
| 3 | Kimi-K3.1-Pro | Claude Code | 61.9 % | 70.0 % | 66.0 % |
| 3 | Grok-4.7 | Grok Build | 61.9 % | 70.0 % | 66.0 % |
| 6 | GPT-6-Astra | Codex | 71.4 % | 60.0 % | 65.7 % |
| 7 | GLM-5.3 | Claude Code | 57.1 % | 60.0 % | 58.5 % |
| 8 | Qwen3.8-Max | Qwen Code | 52.4 % | 50.0 % | … |