Cost & speed
What one passing fix costs at public list price, (across, log scale) against how often each configuration passes (up). Up and to the left is better.
Circle = open-weight model, diamond = proprietary. A hollow marker means fewer than 20 runs (preliminary). Whiskers are 95% Wilson intervals.
Pass rate and cost here pool every tier a configuration ran, and not every configuration ran the same tiers, so compare rows loosely; the scoreboard compares like with like. Each point is one model and harness configuration; the table below names them.
| Model | Harness | Passed/runs | List $ per pass | Input tokens per pass | Output tokens per pass | Median wall time |
|---|---|---|---|---|---|---|
| GLM-5.3 | OpenCode 1.19.3 (neutral) | 90/100 | $0.16 | 1,448,900 | 3,600 | 3 min 00 s |
| GLM-5.3 | OpenCode 1.19.3 (neutral, v2 config) | 9/10 | $0.18 | 1,833,700 | 5,200 | 8 min 30 s |
| DeepSeek V4.1 Flash | OpenCode 1.19.3 (neutral) | 85/104 | $0.43 | 1,958,200 | 8,500 | 2 min 30 s |
| DeepSeek V4.1 Flash | OpenCode 1.19.3 (neutral, v2 config) | 7/10 | $0.62 | 1,893,400 | 10,300 | 3 min 30 s |
| Gemma 4 31B | OpenCode 1.19.3 (neutral) | 22/91 | $0.89 | 755,100 | 2,500 | 5 min 00 s |
| MiniMax M3 | OpenCode 1.19.3 (neutral) | 77/96 | $1.90 | 4,789,700 | 17,900 | 10 min 00 s |
| GPT-6 Astra | Codex CLI (native) | 87/103 | $2.20 | 170,600 | 1,700 | 1 min 00 s |
| MiniMax M3 | OpenCode 1.19.3 (neutral, v2 config) | 9/10 | $2.40 | 7,417,300 | 25,800 | 12 min 30 s |
| Claude Opus 5.5 | OpenCode 1.19.3 (neutral, v2 config) | 10/10 | $3.40 | 815,300 | 7,800 | 2 min 30 s |
| Claude Opus 5.5 | OpenCode 1.19.3 (neutral) | 92/103 | $4.20 | 823,500 | 10,200 | 2 min 00 s |
| DeepSeek V4 Pro | OpenCode 1.19.3 (neutral) | 82/103 | $4.40 | 2,201,300 | 9,700 | 4 min 00 s |
| Kimi K3 | OpenCode 1.19.3 (neutral) | 92/99 | $6.30 | 2,105,300 | 6,600 | 9 min 30 s |
| Claude Opus 5.5 | Claude Code (native) | 95/103 | $7.60 | 1,734,800 | 700 | 1 min 30 s |
| Qwen 3.8 2.4T-A95B | OpenCode 1.19.3 (neutral) | 80/94 | $8.40 | 3,528,000 | 6,400 | 7 min 30 s |
| Kimi K3 | OpenCode 1.19.3 (neutral, v2 config) | 10/10 | $9.50 | 3,286,500 | 9,800 | 12 min 00 s |
| DeepSeek V4 Pro | OpenCode 1.19.3 (neutral, v2 config) | 9/10 | $10.00 | 7,632,400 | 29,700 | 8 min 30 s |
| GPT-6 Astra | OpenCode 1.19.3 (neutral) | 95/103 | $11.00 | 1,017,000 | 4,200 | 2 min 30 s |
| GPT-6 Astra | OpenCode 1.19.3 (neutral, v2 config) | 10/10 | $14.00 | 1,317,700 | 6,700 | 3 min 00 s |
| Qwen 3.8 2.4T-A95B | OpenCode 1.19.3 (neutral, v2 config) | 7/10 | $14.00 | 4,679,500 | 8,300 | 16 min 30 s |
| Claude Fable 5.1 | OpenCode 1.19.3 (neutral) | 27/28 | $27.00 | 2,497,800 | 28,300 | 6 min 00 s |
| Claude Fable 5.1 | Claude Code (native) | 19/21 | $55.00 | 4,644,700 | 2,000 | 4 min 00 s |
| Gemma 4 31B | OpenCode 1.19.3 (neutral, v2 config) | 1/10 | too few passes | – | – | – |
| gpt-oss-120b | OpenCode 1.19.3 (neutral) | 0/99 | too few passes | – | – | – |
| gpt-oss-120b | OpenCode 1.19.3 (neutral, v2 config) | 0/10 | too few passes | – | – | – |
| MiMo V2.6 Pro | OpenCode 1.19.3 (neutral) | 37/90 | unpriced | 1,889,000 | 4,000 | 10 min 00 s |
| MiMo V2.6 Pro | OpenCode 1.19.3 (neutral, v2 config) | 7/10 | unpriced | 2,969,200 | 7,800 | 29 min 30 s |
| Qwen 3.8 Max | OpenCode 1.19.3 (neutral) | 71/94 | unpriced | 2,827,800 | 7,400 | 14 min 00 s |
| Qwen 3.8 Max | OpenCode 1.19.3 (neutral, v2 config) | 7/10 | unpriced | 3,144,200 | 9,600 | 16 min 30 s |
"Unpriced" means the provider publishes no list price for that model. Tokens are rounded to the nearest 100 and wall time to 30 seconds. Input tokens are as the provider reported them, including cached reads. Wall time covers the whole run (setup, the agent, the checks and the review), and runs ran at varying concurrency, so it is indicative only. List-price figures are not anyone's invoice. Cost and reliability count every attempt, including re-runs of a case, so their run counts can be slightly higher than the matrix's.