Harness effect
The same model, run in the neutral harness and in its own lab's harness, counted only on the cases that ran in both, so the two rows share exactly the same cases. Overlapping bars mean this evidence does not separate the two harnesses; it does not prove they are equal.
Circle = open-weight model, diamond = proprietary. A hollow marker means fewer than 20 runs (preliminary). Whiskers are 95% Wilson intervals.
Solid whisker: neutral harness. Dashed whisker: the lab's own harness.
| Model | Harness | Pass rate | 95% interval | Passed/runs |
|---|---|---|---|---|
| Claude Opus 5.5 | OpenCode 1.19.3 (neutral) | 89% | 82%–94% | 92/103 |
| Claude Opus 5.5 | Claude Code (native) | 92% | 85%–96% | 95/103 |
| GPT-6 Astra | OpenCode 1.19.3 (neutral) | 92% | 85%–96% | 95/103 |
| GPT-6 Astra | Codex CLI (native) | 84% | 76%–90% | 87/103 |
| Claude Fable 5.1 | OpenCode 1.19.3 (neutral) | 95% | 76%–99% | 19/20 |
| Claude Fable 5.1 | Claude Code (native) | 90% | 70%–97% | 18/20 |
Speed and cost differ by harness too; see cost & speed, where each harness has its own row.