elwood.sh $

Scoreboard

Pass rate by model and harness, per difficulty tier. Sorted by the lower end of the 95% interval. A tier appears once it has scored runs.

Circle = open-weight model, diamond = proprietary. A hollow marker means fewer than 20 runs (preliminary). Whiskers are 95% Wilson intervals.

Tier 3 (hard)

0%25%50%75%100%Kimi K3 · neutralClaude Opus 5.5 · Claude CodeGPT-6 Astra · neutralClaude Fable 5.1 · neutralGLM-5.3 · neutralClaude Opus 5.5 · neutral v2GPT-6 Astra · neutral v2Kimi K3 · neutral v2Claude Fable 5.1 · Claude CodeClaude Opus 5.5 · neutralDeepSeek V4.1 Flash · neutralGPT-6 Astra · CodexQwen 3.8 2.4T-A95B · neutralMiniMax M3 · neutralDeepSeek V4 Pro · neutral v2GLM-5.3 · neutral v2MiniMax M3 · neutral v2Qwen 3.8 Max · neutralDeepSeek V4 Pro · neutralDeepSeek V4.1 Flash · neutral v2MiMo V2.6 Pro · neutral v2Qwen 3.8 2.4T-A95B · neutral v2Qwen 3.8 Max · neutral v2MiMo V2.6 Pro · neutralGemma 4 31B · neutralGemma 4 31B · neutral v2gpt-oss-120b · neutralgpt-oss-120b · neutral v2

Shaded rows are not separable from the leader on this evidence: their interval reaches the lower end of the leader (the solid result with the highest lower end). Preliminary rows never lead and are never shaded.

Tier 3 (hard), as a table
ModelHarnessWeightsPass rate95% intervalPassed/runsNotes
Kimi K3OpenCode 1.19.3 (neutral)open-weight98%88%–100%43/44top group
Claude Opus 5.5Claude Code (native)proprietary91%79%–96%40/44top group
GPT-6 AstraOpenCode 1.19.3 (neutral)proprietary91%79%–96%40/44top group
Claude Fable 5.1OpenCode 1.19.3 (neutral)proprietary95%76%–99%19/20top group paused
GLM-5.3OpenCode 1.19.3 (neutral)open-weight89%76%–95%39/44top group
Claude Opus 5.5OpenCode 1.19.3 (neutral, v2 config)proprietary100%72%–100%10/10preliminary
GPT-6 AstraOpenCode 1.19.3 (neutral, v2 config)proprietary100%72%–100%10/10preliminary
Kimi K3OpenCode 1.19.3 (neutral, v2 config)open-weight100%72%–100%10/10preliminary
Claude Fable 5.1Claude Code (native)proprietary90%71%–97%19/21top group paused
Claude Opus 5.5OpenCode 1.19.3 (neutral)proprietary82%68%–90%36/44top group
DeepSeek V4.1 FlashOpenCode 1.19.3 (neutral)open-weight80%66%–89%35/44top group
GPT-6 AstraCodex CLI (native)proprietary80%66%–89%35/44top group
Qwen 3.8 2.4T-A95BOpenCode 1.19.3 (neutral)open-weight78%63%–88%32/41
MiniMax M3OpenCode 1.19.3 (neutral)open-weight77%62%–87%33/43
DeepSeek V4 ProOpenCode 1.19.3 (neutral, v2 config)open-weight90%60%–98%9/10preliminary
GLM-5.3OpenCode 1.19.3 (neutral, v2 config)open-weight90%60%–98%9/10preliminary
MiniMax M3OpenCode 1.19.3 (neutral, v2 config)open-weight90%60%–98%9/10preliminary
Qwen 3.8 MaxOpenCode 1.19.3 (neutral)open-weight73%58%–84%32/44
DeepSeek V4 ProOpenCode 1.19.3 (neutral)open-weight66%51%–78%29/44
DeepSeek V4.1 FlashOpenCode 1.19.3 (neutral, v2 config)open-weight70%40%–89%7/10preliminary
MiMo V2.6 ProOpenCode 1.19.3 (neutral, v2 config)open-weight70%40%–89%7/10preliminary
Qwen 3.8 2.4T-A95BOpenCode 1.19.3 (neutral, v2 config)open-weight70%40%–89%7/10preliminary
Qwen 3.8 MaxOpenCode 1.19.3 (neutral, v2 config)open-weight70%40%–89%7/10preliminary
MiMo V2.6 ProOpenCode 1.19.3 (neutral)open-weight21%12%–36%9/42
Gemma 4 31BOpenCode 1.19.3 (neutral)open-weight17%8%–31%7/41
Gemma 4 31BOpenCode 1.19.3 (neutral, v2 config)open-weight10%2%–40%1/10preliminary
gpt-oss-120bOpenCode 1.19.3 (neutral)open-weight0%0%–8%0/43known format defect
gpt-oss-120bOpenCode 1.19.3 (neutral, v2 config)open-weight0%0%–28%0/10preliminary known format defect

Tier 2 (moderate)

0%25%50%75%100%Claude Opus 5.5 · neutralGPT-6 Astra · neutralGPT-6 Astra · CodexClaude Opus 5.5 · Claude CodeDeepSeek V4 Pro · neutralQwen 3.8 2.4T-A95B · neutralGLM-5.3 · neutralDeepSeek V4.1 Flash · neutralKimi K3 · neutralMiniMax M3 · neutralQwen 3.8 Max · neutralMiMo V2.6 Pro · neutralGemma 4 31B · neutralgpt-oss-120b · neutral

Shaded rows are not separable from the leader on this evidence: their interval reaches the lower end of the leader (the solid result with the highest lower end). Preliminary rows never lead and are never shaded.

Tier 2 (moderate), as a table
ModelHarnessWeightsPass rate95% intervalPassed/runsNotes
Claude Opus 5.5OpenCode 1.19.3 (neutral)proprietary96%82%–99%27/28top group
GPT-6 AstraOpenCode 1.19.3 (neutral)proprietary96%82%–99%27/28top group
GPT-6 AstraCodex CLI (native)proprietary96%82%–99%27/28top group
Claude Opus 5.5Claude Code (native)proprietary93%77%–98%26/28top group
DeepSeek V4 ProOpenCode 1.19.3 (neutral)open-weight93%77%–98%26/28top group
Qwen 3.8 2.4T-A95BOpenCode 1.19.3 (neutral)open-weight93%77%–98%26/28top group
GLM-5.3OpenCode 1.19.3 (neutral)open-weight89%73%–96%25/28top group
DeepSeek V4.1 FlashOpenCode 1.19.3 (neutral)open-weight86%68%–94%24/28top group
Kimi K3OpenCode 1.19.3 (neutral)open-weight86%68%–94%24/28top group
MiniMax M3OpenCode 1.19.3 (neutral)open-weight82%64%–92%23/28top group
Qwen 3.8 MaxOpenCode 1.19.3 (neutral)open-weight79%60%–90%22/28top group
MiMo V2.6 ProOpenCode 1.19.3 (neutral)open-weight54%36%–70%15/28
Gemma 4 31BOpenCode 1.19.3 (neutral)open-weight14%6%–32%4/28
gpt-oss-120bOpenCode 1.19.3 (neutral)open-weight0%0%–12%0/28known format defect

Tier 1 (routine)

0%25%50%75%100%GLM-5.3 · neutralKimi K3 · neutralClaude Opus 5.5 · neutralClaude Opus 5.5 · Claude CodeGPT-6 Astra · neutralDeepSeek V4 Pro · neutralQwen 3.8 2.4T-A95B · neutralDeepSeek V4.1 Flash · neutralMiniMax M3 · neutralGPT-6 Astra · CodexQwen 3.8 Max · neutralMiMo V2.6 Pro · neutralGemma 4 31B · neutralgpt-oss-120b · neutral

Shaded rows are not separable from the leader on this evidence: their interval reaches the lower end of the leader (the solid result with the highest lower end). Preliminary rows never lead and are never shaded.

Tier 1 (routine), as a table
ModelHarnessWeightsPass rate95% intervalPassed/runsNotes
GLM-5.3OpenCode 1.19.3 (neutral)open-weight96%82%–99%26/27top group
Kimi K3OpenCode 1.19.3 (neutral)open-weight96%81%–99%25/26top group
Claude Opus 5.5OpenCode 1.19.3 (neutral)proprietary94%79%–98%29/31top group
Claude Opus 5.5Claude Code (native)proprietary94%79%–98%29/31top group
GPT-6 AstraOpenCode 1.19.3 (neutral)proprietary90%75%–97%28/31top group
DeepSeek V4 ProOpenCode 1.19.3 (neutral)open-weight87%70%–95%26/30top group
Qwen 3.8 2.4T-A95BOpenCode 1.19.3 (neutral)open-weight88%70%–96%22/25top group
DeepSeek V4.1 FlashOpenCode 1.19.3 (neutral)open-weight84%67%–93%26/31top group
MiniMax M3OpenCode 1.19.3 (neutral)open-weight84%65%–94%21/25top group
GPT-6 AstraCodex CLI (native)proprietary81%64%–91%25/31top group
Qwen 3.8 MaxOpenCode 1.19.3 (neutral)open-weight81%60%–92%17/21top group
MiMo V2.6 ProOpenCode 1.19.3 (neutral)open-weight65%43%–82%13/20top group
Gemma 4 31BOpenCode 1.19.3 (neutral)open-weight50%31%–69%11/22
gpt-oss-120bOpenCode 1.19.3 (neutral)open-weight0%0%–12%0/28known format defect
Claude Fable 5.1OpenCode 1.19.3 (neutral)proprietarytoo few runs to raten=8paused