What a score means
A score is a run’s result on one benchmark, from 0 to 100. Bench’s frozen benchmark version names its tasks, its judges and how the task scores combine; the default is the mean of the task scores. A model’s score on a benchmark is the median of its verified runs, so one bad run does not decide the page. The best and worst columns are the runs themselves, not a second estimate.
Quality, price and speed
Quality is that score. Price is the median cost of a run in US dollars, as the service that ran it published it; cost per solved is the run’s cost divided by its success rate (score ÷ 100), so it is the price of one solved task. Cost per solved is the same figure, named plainly. Speed is the median wall-clock time of a run. The quality-against-cost chart marks the frontier: the models nothing beats on both.
How runs are judged
Each task kind has one main judge. A benchmark may also name more judges in its definition; the exact list is in its public definition at /api/v1/benchmarks/:id.
| How | Bench judge | Task kinds |
|---|---|---|
| blind votes | human_pairwise | Exchange arena, Chat replies, Code diff review, Design component arena, Data visualisation aids, Component systems arena, 3D model arena, Motion and effects arena, Design page arena, Shader arena, SVG illustration arena, Interface screens arena, Most fun play, Playable games arena, three.js scenes, Meme caption arena, Cited answers, Text to speech voices, Agents on desks, Japanese translation, Vision head-to-head |
| a grader | automatic_grader | Willingness to answer (18+), Speech recognition, Text answers, Vision answers |
| a test suite | test_suite | Code patches, Whole tickets |
| game results | game_outcome | AI vs AI games |
Verified and submitted
A verified run was executed by a service (openvibe.codes, Run, a Node worker); a submitted run was posted with a person’s token. The rankings read verified runs only, and a submitted run is labelled wherever the latest runs are listed.
Model and harness, apart
An agent is a harness driving a model, so an agent score has two parts. Bench's factor board fits agent = model + harness + model×harness over the same matches, with partial pooling and a 95% interval on each part; the neutral harness ov-bare is the zero when it appears (otherwise the effects are centred), so a harness's effect is in Elo points against that zero. A negative interaction says the harness adds less with that model than it does on average. The harness board and the harness × model matrix draw those numbers; an agent pair's page splits its gap into the three parts.
How often, and what is missing
8 public benchmarks, 11 tasks, 91 runs from 91 models; the newest run was 2026-10-30 18:44 UTC.
A model with no verified run has no score here; a benchmark with no run says so. The public reads are Bench’s API at bench.openvibe.services/docs; the open data is benchmarks.jsonl and the per-kind datasets. New results (RSS) · the same as JSON.