How we evaluate

Each test is a real bug that was fixed once before, replayed so a model has to fix it again. Passing takes a working, committed fix that an independent reviewer accepts.

One run, step by step

A replayed bug. The code is reset to just before a real fix was made. The model gets the bug report, written the way an engineer would hand it over, and nothing else.
A planted failing test. The harness adds a test that fails on the buggy code and passes on a correct fix. The model can run it, but changing or weakening it is caught and counted.
It must commit. The deliverable is a commit. A run that edits files but never commits, or never edits at all, fails.
The checks run. The planted test and the project's own build and regression checks must pass.
An independent review. A separate reviewer model reads the diff and must approve it. A fix that passes the test but games it, for example by special-casing the test, is blocked.

The same harness for everyone

Every model runs headless in the same engine, OpenCode 1.19.3 (the engine Elwood is built on), with the same tools and permissions, and the same time limit on the agent for a given case. Elwood's rebrand changes nothing a model sees: the instructions and tool descriptions sent to the model are byte-for-byte the same as OpenCode's. A second configuration, "v2", tunes the harness configuration for each model (for example, prompt caching where the model supports it). It is reported as its own row, never mixed in. So far v2 has run on a fixed set of 10 hard cases, not a random sample, so a v2 row is not directly comparable with a v1 row.

Some models also run in their own lab's harness: Claude in Claude Code and GPT in Codex. Comparing the same model in both, on the harness effect page, separates the model's skill from the harness's.

Reading the numbers

Pass rate with a 95% interval. Intervals are Wilson score intervals. The scoreboard sorts by the lower end of the interval (a conservative ranking). The leader is the result with the highest lower end among rows with 20 or more runs, and a row is marked "top group" when its interval reaches the leader's lower end. That is a screen, not a test: it means this evidence does not separate the row from the leader, not that the two are equal. Preliminary rows are never marked. Sorting and marking use the unrounded bounds.
Sample size. A rate is shown only with at least 10 runs. Under 20 runs, the marker is hollow and labelled preliminary.
Cost per passing fix, at public list price. Total list-price spend divided by the number of passing runs, so failed attempts count toward the cost of each success. Shown only with at least 5 passes. It is not what anyone actually paid.
Wall time is the whole run. It covers setting up the case, the agent's work, the checks and the review, so it can be longer than the agent's own time limit.
Every rate comes from the published matrix. The scoreboard and the harness comparison are computed from the same per-case results that data.json publishes, so anyone can re-derive them.
Invalidated runs are excluded. When a run was spoiled by something other than the model (a worker restart, an exhausted budget, a provider outage, or a fault in the case's own setup), it is marked invalid and re-run where possible. It never counts for or against a model. 62 runs are currently excluded this way.

Limits, plainly

The cases come from a small number of private production services, mostly TypeScript, so results lean toward that kind of code. They are realistic, not representative of all software.
One open-weight model, gpt-oss-120b, scores zero because of a format problem: it writes its tool calls into its reasoning text, so the harness never sees them. That reflects the model's output format on this provider, not its coding ability, and it is marked as a known defect.
Runs ran at varying concurrency, so wall times are indicative only and are rounded to 30 seconds.
Bug descriptions, code and repository names are never published, because the bugs come from private code.