elwood.sh $

Cases

Every case, with which configurations solved it. Cases are shown only as opaque numbers: the bugs come from private code, so their descriptions are never published. Only cases with at least one scored run are listed.

101224246678101110121912173141618212011222124number of configurations that solved the case
Cases by number of configurations that solved them
Solved byCases
01
10
21
30
42
52
64
76
87
910
1011
1112
1219
1317
143
150
160
170
180
192
201
211
221
232
241

The full matrix

✓ passed, ✗ failed (including refusals), · not run. Scroll sideways inside the table on small screens.

CaseTierClaude Fable 5.1 · Claude CodeClaude Opus 5.5 · Claude CodeGPT-6 Astra · CodexClaude Fable 5.1 · neutralClaude Opus 5.5 · neutralDeepSeek V4 Pro · neutralDeepSeek V4.1 Flash · neutralGLM-5.3 · neutralGPT-6 Astra · neutralGemma 4 31B · neutralKimi K3 · neutralMiMo V2.6 Pro · neutralMiniMax M3 · neutralQwen 3.8 2.4T-A95B · neutralQwen 3.8 Max · neutralgpt-oss-120b · neutralClaude Opus 5.5 · neutral v2DeepSeek V4 Pro · neutral v2DeepSeek V4.1 Flash · neutral v2GLM-5.3 · neutral v2GPT-6 Astra · neutral v2Gemma 4 31B · neutral v2Kimi K3 · neutral v2MiMo V2.6 Pro · neutral v2MiniMax M3 · neutral v2Qwen 3.8 2.4T-A95B · neutral v2Qwen 3.8 Max · neutral v2gpt-oss-120b · neutral v2
CASE-001Tier 1·✓✓✓✓✓✓✓✓✓✓✗✓✓✓✗············
CASE-002Tier 1·✓✓✓✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-003Tier 1·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-004Tier 1·✓✓·✓✓✓✓✓✓✓✓✓✓✓✗············
CASE-005Tier 1·✓✓·✓✓✓✓✓·✓··✓·✗············
CASE-006Tier 1·✓✓·✓✓✗✓✓✓✓·✓✓·✗············
CASE-007Tier 1·✓✓✓✓✓✓✓✓✓✓·✗✓✓✗············
CASE-008Tier 1·✓✓·✓✓✓·✓···················
CASE-009Tier 1·✓✓·✓✓✓✓✓✓✓✓✓✓✓✗············
CASE-010Tier 1·✓✓·✓✓✓✓✓✓✓✓✓✓✓✗············
CASE-011Tier 1·✓✗·✓·✓·✓······✗············
CASE-012Tier 1·✓✗·✓✓✗·✓······✗············
CASE-013Tier 1·✗✗·✓✗✓✓✓·✓·✓✗·✗············
CASE-014Tier 1·✓✓·✓✓✓✓✓✓✓✗✓✓✓✗············
CASE-015Tier 1·✗✗·✗✗✗✗✗✗✗✗✗✗✗✗············
CASE-016Tier 1·✓✓·✓✓✓✓✓✗✓·✓✓··············
CASE-017Tier 1·✓✗·✓✓✓✓✗✗✓✓✓✓✓✗············
CASE-018Tier 1·✓✓✓✗✗✗✓✓✗✓✗✗✗✗✗············
CASE-019Tier 1·✓✓✓✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-020Tier 1·✓✓✓✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-021Tier 1·✓✓·✓✓✓✓✓✓✓✓✓✓✓✗············
CASE-022Tier 1·✓✓·✓✓✓✓✓✗✓·✓·✓✗············
CASE-023Tier 1·✓✓·✓✓✓✓✓✓✓✓✓✓✓✗············
CASE-024Tier 1·✓✗·✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-025Tier 1·✓✓·✓✗✓✓✓✓✓✓✓✓✓✗············
CASE-026Tier 1·✓✓·✓✓✓✓✓·✓✓✓✓·✗············
CASE-027Tier 1·✓✓✓✓✓✓✓✗✗✓✗✗✓✗✗············
CASE-028Tier 1·✓✓·✓✓✗·✓······✗············
CASE-029Tier 1·✓✓·✓✓✓✓✓·✓✓✓✓✓✗············
CASE-030Tier 1·✓✓·✓✓✓✓✓···················
CASE-031Tier 1·✓✓✓✓✓✓✓✓✓✓✓✓✓✗✗············
CASE-032Tier 2·✓✓·✓✓✗✓✓✗✓✗✓✓✗✗············
CASE-033Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-034Tier 2·✓✓·✓✓✓✓✓✓✓✗✓✓✗✗············
CASE-035Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-036Tier 2·✓✓·✓✗✓✓✓✗✗✗✗✓✓✗············
CASE-037Tier 2·✗✗·✗✓✗✗✗✗✗✗✓✗✗✗············
CASE-038Tier 2·✓✓·✓✓✗✓✓✗✓✓✓✓✓✗············
CASE-039Tier 2·✓✓·✓✓✗✓✓✗✗✗✗✗✓✗············
CASE-040Tier 2·✓✓·✓✓✓✗✓✗✓✓✓✓✓✗············
CASE-041Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-042Tier 2·✓✓·✓✓✓✓✓✗✗✗✗✓✗✗············
CASE-043Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-044Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-045Tier 2·✓✓·✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-046Tier 2·✓✓·✓✓✓✓✓✓✓✓✓✓✓✗············
CASE-047Tier 2·✓✓·✓✗✓✗✓✗✓✗✓✓✗✗············
CASE-048Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-049Tier 2·✓✓·✓✓✓✓✓✓✓✓✓✓✓✗············
CASE-050Tier 2·✓✓·✓✓✓✓✓✗✓✗✓✓✗✗············
CASE-051Tier 2·✓✓·✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-052Tier 2·✓✓·✓✓✓✓✓✗✓✗✗✓✓✗············
CASE-053Tier 2·✓✓·✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-054Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-055Tier 2·✗✓·✓✓✓✓✓✗✓✗✗✓✓✗············
CASE-056Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-057Tier 2·✓✓·✓✓✓✓✓✓✓✓✓✓✓✗············
CASE-058Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-059Tier 2·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-060Tier 3·✓✓·✓✓✓✓✓✗✓✗✗✓✓✗············
CASE-061Tier 3✓✓✓✓✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-062Tier 3·✓✓·✓✓✗✓✗✓✓✗✓✓✗✗············
CASE-063Tier 3·✓✓·✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-064Tier 3·✗✓·✗✗✓✓✓✗✓✓✓✓✓✗············
CASE-065Tier 3✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-066Tier 3·✓✓·✓✗✓✗✓✗✗✗✓✓✗✗············
CASE-067Tier 3·✓✗·✓✗✓✓✓✓✓✗✓✗✓✗············
CASE-068Tier 3✓✓✓✓✓✓✓✓✓✗✓✗✓✓✓✗✓✓✓✓✓✗✓✓✓✓✓✗
CASE-069Tier 3·✓✓·✓✓✓✓✓✗✓✗✗✗✗✗············
CASE-070Tier 3·✓✓·✗✗✗✓✗✗✓✗✓✓✓✗············
CASE-071Tier 3·✓✗·✗✗✓✗✓✗✓✗✗✓✗✗············
CASE-072Tier 3✓✓✓✓✗✓✓✓✓·✓✗✓✗✓✗✓✓✓✓✓✗✓✗✓✓✗✗
CASE-073Tier 3·✓✗·✗✓✓✓✓✗✓✗✓✓✓✗············
CASE-074Tier 3✓✓✓✓✓✓✗✓✓✗✓✗✗✗✗✗✓✗✗✗✓✗✓✗✓✗✗✗
CASE-075Tier 3✓✓✓✓✓✓✓✓✓·✓···✓·············
CASE-076Tier 3✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓✗✓✓✓✓✓✗✓✓✓✓✓✗
CASE-077Tier 3✗✓✓✓✓✗✓✓✓✗✓✗✗✓✓✗✓✓✓✓✓✗✓✓✓✓✓✗
CASE-078Tier 3✓✓✓✓✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-079Tier 3✓✗✓✓✓✓✓✓✓✗✓✓✓✗✓✗············
CASE-080Tier 3·✓✓·✗✓✓✓✓✗✓✗✗✗✗✗············
CASE-081Tier 3·✗✓·✗✓✗✓✓✗✓✓✗✓✓✗············
CASE-082Tier 3·✓✓·✓✗✓✓✓✗✓✗✓✓✓✗············
CASE-083Tier 3·✓✓·✓✓✓✓✓✗✓✗✓✓✓✗············
CASE-084Tier 3✓✓✓✓✓✓✓✓✓✓✓✗✓✓✗✗✓✓✓✓✓✗✓✓✓✓✓✗
CASE-085Tier 3✓✓✓✓✓✗✓✓✓✗✓✗✗✗✓✗✓✓✗✓✓✗✓✓✓✓✓✗
CASE-086Tier 3✓✓✓✓✓✓✓✓✓✓✓✗✓✓✓✗············
CASE-087Tier 3✓✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-088Tier 3✓✓✗✓✓✗✓✓✓✗✓✗✓✓✓✗············
CASE-089Tier 3✓✓✗✓✗✗✗✓✗✗✓✗✓✓✓✗············
CASE-090Tier 3·✓✗·✓✓✓✓✓✓✓✗✓✗✗✗············
CASE-091Tier 3·✓✓·✓✓✓✓✓✗✓✗✓✓✗✗············
CASE-092Tier 3✓✓✓✓✓✓✓✗✓✗✓✗✓✓✓✗✓✓✓✓✓✓✓✓✓✗✗✗
CASE-093Tier 3·✓✓·✓✓✓✓✓✗✓✗✓✓✗✗············
CASE-094Tier 3·✓✓·✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-095Tier 3✓✓✓✓✓✗✓✓✓✗✓✗✓✓✓✗············
CASE-096Tier 3·✓✗·✓✗✗✗✓✗✓✓✓✗✗✗············
CASE-097Tier 3·✓✗·✓✗✓✗✓✗✓✗✗✓✓✗············
CASE-098Tier 3✓✓✓✓✓✓✓✓✓·✓·✓·✓✗✓✓✓✓✓✗✓✓✓✓✓✗
CASE-099Tier 3·✗✓·✓✓✗✓✗✓✓✗✓✓✗✗············
CASE-100Tier 3·✓✗·✓✗✓✓✓✗✓✗✓✓✓✗············
CASE-101Tier 3·✓✓·✓✓✗✓✓✓✓✗✓✓✓✗············
CASE-102Tier 3✓✓✓✓✓✓✓✓✓✗✓✓✓✓✓✗············
CASE-103Tier 3✗✓✓✗✓✗✗✓✓✗✓✗✗·✓✗✓✓✗✓✓✗✓✗✗✗✓✗