Reliability
How often a configuration does something other than honestly fail or succeed. Percentages are of all scored runs. The categories overlap: one run can, for example, both go silent and be refused, so rows need not add up.
- Claimed done, wasn't: the agent said the fix was finished, but it did not pass.
- Weakened tests: the agent changed or deleted tests instead of fixing the code.
- Went silent: the session ended with no fix and no explanation.
- Blocked by review: the independent reviewer refused the fix.
- Refusals: a count of runs, not a percentage, where the model declined the task.
| Model | Harness | Runs | Claimed done, wasn't | Weakened tests | Went silent | Blocked by review | Refusals (runs) |
|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | OpenCode 1.19.3 (neutral, v2 config) | 10 | 10% | 0% | 20% | 10% | 0 |
| DeepSeek V4.1 Flash | OpenCode 1.19.3 (neutral) | 104 | 5.8% | 0% | 13.5% | 0% | 0 |
| Claude Fable 5.1 | OpenCode 1.19.3 (neutral) | 28 | 3.6% | 0% | 3.6% | 0% | 1 |
| DeepSeek V4 Pro | OpenCode 1.19.3 (neutral) | 103 | 2.9% | 0% | 2.9% | 8.7% | 0 |
| MiMo V2.6 Pro | OpenCode 1.19.3 (neutral) | 90 | 2.2% | 0% | 52.2% | 0% | 5 |
| Qwen 3.8 2.4T-A95B | OpenCode 1.19.3 (neutral) | 94 | 1.1% | 0% | 9.6% | 1.1% | 6 |
| Qwen 3.8 Max | OpenCode 1.19.3 (neutral) | 94 | 1.1% | 0% | 20.2% | 1.1% | 0 |
| GPT-6 Astra | OpenCode 1.19.3 (neutral) | 103 | 1% | 0% | 1% | 1% | 3 |
| gpt-oss-120b | OpenCode 1.19.3 (neutral) | 99 | 1% | 0% | 100% | 0% | 4 |
| Claude Fable 5.1 | Claude Code (native) | 21 | 0% | 0% | 0% | 4.8% | 0 |
| Claude Opus 5.5 | OpenCode 1.19.3 (neutral) | 103 | 0% | 0% | 2.9% | 2.9% | 3 |
| Claude Opus 5.5 | Claude Code (native) | 103 | 0% | 0% | 0% | 2.9% | 0 |
| Claude Opus 5.5 | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 0% | 0% | 0 |
| DeepSeek V4 Pro | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 10% | 0% | 0 |
| Gemma 4 31B | OpenCode 1.19.3 (neutral) | 91 | 0% | 0% | 27.5% | 4.4% | 6 |
| Gemma 4 31B | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 30% | 10% | 0 |
| GLM-5.3 | OpenCode 1.19.3 (neutral) | 100 | 0% | 0% | 1% | 2% | 0 |
| GLM-5.3 | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 10% | 0% | 0 |
| GPT-6 Astra | Codex CLI (native) | 103 | 0% | 0% | 0% | 6.8% | 0 |
| GPT-6 Astra | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 0% | 0% | 0 |
| gpt-oss-120b | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 100% | 0% | 0 |
| Kimi K3 | OpenCode 1.19.3 (neutral) | 99 | 0% | 0% | 4% | 0% | 0 |
| Kimi K3 | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 0% | 0% | 0 |
| MiMo V2.6 Pro | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 30% | 0% | 0 |
| MiniMax M3 | OpenCode 1.19.3 (neutral) | 96 | 0% | 0% | 14.6% | 3.1% | 4 |
| MiniMax M3 | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 10% | 0% | 0 |
| Qwen 3.8 2.4T-A95B | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 20% | 10% | 0 |
| Qwen 3.8 Max | OpenCode 1.19.3 (neutral, v2 config) | 10 | 0% | 0% | 30% | 0% | 0 |
Only configurations with at least 10 scored runs are listed.