Prism Arena Bench a sweep
Every contender receives a byte-identical prompt and answers with one JSON scene spec. The arena validates it against a zod contract, renders it in WebGL, animates it on an externally clocked timeline, and a blind four-judge panel scores the result out of 20.
A mean of means, in that order. Judgment rows are never pooled into one flat average — that would silently weight tasks by how many judgments happen to exist.
01 Standings
Headline score is the mean across tasks of each task's panel mean. Beside it sits the per-task filmstrip that produced it and the two validity rates, because a rank is not a finding on its own.
02 Reliability against peak
Per-attempt validity counts every generation. Best-of-N validity counts a task as covered if any attempt parsed. Best-of-N is strictly the more flattering number, so both are shown: the gap between them is how much retries rescued.
03 The full matrix
One cell per model/task pair. Lightness encodes the task panel mean; a hatched cell is a pair with no score at all. This table is also the text equivalent of the filmstrips in the standings.
04 Task difficulty
Each strip plots one tick per model on the shared 0–20 track. A wide strip is a task that discriminates. A task where everyone clusters at the top has saturated and is no longer buying information, however elegant its prompt.
05 Rubric shape
Four dimensions, 0–5 each, averaged over the panel and then over tasks. All eleven plots share identical axes and rings, so the silhouettes are directly comparable.
06 Judge disagreement
An averaged number with hidden spread invites false confidence. Every panel mean on this page is the middle of a range, and the range is worth seeing.
07 In-depth analysis
The score is a systems result: contract compliance, execution, rendering, motion, and blind rubric judgment. The evidence below reads those layers together.
08 Noticeable findings
These observations had the largest consequences for the published result, from evidence drift to provider and specification failures.
09 Pain points
Operational friction is part of the measurement. These are the failure surfaces that still make a full sweep slower, costlier, or harder to reproduce.
10 Roadmap
More attempts, stronger judge controls, human grounding, and deeper tasks turn this first arena into a benchmark whose close calls can carry statistical weight.
11 Method and limits
Every contender gets the same task prompt concatenated with the same shared
OUTPUT_CONTRACT, through the same command. Only --model and
--thinking differ between contenders. Nothing is tuned per model —
no bespoke system prompt, no per-vendor formatting hint, no retry-until-parses loop.
The runner passes --no-tools --no-skills --no-rules. Several roster
models are agentic coding models: handed a write tool, they satisfy the task by
writing the spec to a file and replying with a prose summary, which the validator
scores as malformed. That penalises a channel mismatch rather than 3D reasoning.
One qwen response went from 390 bytes of prose to 10 KB of real JSON once tools were
disabled — same model, same prompt, same rubric.
Each model/task cell runs two attempts. The judged attempt is the first one that validates, chosen by validity and then by start time — never by quality. Ranking attempts by quality would require the judge we are economising on, and would leak the outcome into the selection. Call this best-of-N by validity; it is not the best run.
Per-attempt validity is valid generations over all generations. Best-of-N validity is tasks with at least one valid attempt over all tasks. Best-of-N is strictly more flattering and hides the reliability signal, so the page never shows one without the other.
Judges see redacted submissions under anonymous labels, four per call, against an absolute rubric rather than a ranking. Redaction is structure-aware and never rewrites object ids, so a spec stays coherent while its author stays hidden. Nothing here is decided by vote, which is why the panel does not need to be odd.
Judges receive rendered frames alongside the JSON. If any contender in a judged batch is missing a frame, the whole batch drops to JSON only. No model is ever judged text-only while its peers in the same call are judged with images.
Every judge is also a contender, and the panel spans three families: two Anthropic judges, one OpenAI judge, and one Google judge. Blinding hides identity and family-diverse averaging dilutes any single model's bias, but neither removes self-preference. The judge-family cross-tab above makes that risk inspectable.
A missing cell is drawn as a hatched slot and written as an em dash. A genuine zero is drawn as a bar at the floor and written as 0.00. They are different findings and the page never conflates them: a model that was never run does not get credited with a failure it did not have, and a model that scored nothing does not get hidden behind a gap.
12 References
The benchmark sits beside spatial reasoning, executable generation, structured output, and LLM-as-a-judge research. Each citation states why it matters here.
13 Authors
Prism Arena Bench is an Agentic Engineering research project. Authorship and organisational context are published with the evidence.