{ "$comment": [ "Editorial content and release-snapshot facts for the published Prism Arena Bench page.", "", "results.json remains the source of truth for benchmark numbers. export-results.mjs rewrites", "the corpus block below from results.metadata.counts on every canonical export, and the", "publication verifier rejects a mismatch. The remaining fields are human-authored prose:", "analysis, findings, pain points, roadmap, references, and authorship.", "", "Every reference below was fetched and its title matched against the citation before", "it was added. Do not add a reference you have not resolved." ], "schemaVersion": 1, "corpus": { "rosterModels": 11, "tasks": 10, "runs": 220, "validRuns": 203, "invalidRuns": 17, "judgments": 440, "currentVersionJudgments": 440, "expectedModelTaskPairs": 110, "modelTaskPairsWithRuns": 110, "fullyJudgedModelTaskPairs": 110 }, "title": "Prism Arena Bench", "subtitle": "How eleven language models build and animate the same ten 3D tasks", "authors": [ { "fullName": "Eduardo Javier García López", "role": "Chief Executive Officer, Cofounder", "affiliation": "Agentic Engineering", "email": "eduardo@agenticengineering.agency" }, { "fullName": "Luis Fernando Ramos", "role": "Cofounder — Strategy, Delivery, Sales, Capital", "affiliation": "Agentic Engineering" }, { "fullName": "Ricardo Soberanis", "role": "Cofounder — Consumer Products", "affiliation": "Agentic Engineering" }, { "fullName": "Sebastián García-Moreno Zinchenko", "role": "Cofounder — Product", "affiliation": "Agentic Engineering" } ], "organization": { "name": "Agentic Engineering", "url": "https://agenticengineering.agency", "tagline": "Inspectable AI systems for expensive, recurring workflows.", "description": "Agentic Engineering designs and ships reliable AI agent systems for ops-heavy teams. We map one costly workflow, build the smallest credible production system, and add evals, observability, human approval, and handoff from day one." }, "analysis": { "positioning": "Prism Arena is a systems benchmark for constrained, executable 3D scene-and-motion synthesis by general-purpose language models. The unit of evaluation is the whole pipeline: one natural-language task pack becomes SceneSpec v1 JSON, a single React Three Fiber runtime materialises every submission, Anime.js supplies the only motion, and a label-blinded panel scores the available evidence against a fixed four-dimension, twenty-point rubric. Four judges made 120 batch invocations, yielding 440 contender-level rows. Complete batches include rendered frames; an incomplete batch falls back uniformly to JSON-only evidence. That places Prism between spatial-reasoning benchmarks that ask a model to read a scene and generative benchmarks that ask it to make one, while borrowing the executable-verification discipline of code benchmarks: structural validity and rendered quality remain separately inspectable.", "sections": [ { "heading": "What the score actually measures", "body": "A headline score is the panel-mean total across ten task packs, each scored out of twenty. SceneSpec v1 validates geometry, materials, lights, an ordered timeline of two to eight phases, and the offset-plus-span arithmetic that keeps an action inside its phase. Validity is not a rubric dimension. Because each cell has two recorded runs, the selector sends the earliest valid run to the panel; if neither validates, it sends the earliest invalid run rather than assigning an automatic score. The sole double-invalid cell in this corpus received zero from all four judges. The headline therefore describes the validity-selected artifact, while the separate per-recorded-run validity rate exposes how reliably a model meets the contract." }, { "heading": "Validity and quality are different abilities", "body": "The roster separates cleanly on schema compliance in a way that does not track model prestige. Three models never emitted an invalid spec across twenty recorded runs; the weakest managed exactly three quarters. The common failure is not malformed JSON but omitted required fields — a material without its emissive intensity, an object without its shadow flag — which is a specification-following failure rather than a syntax failure. This is the same distinction JSONSchemaBench draws, and it argues for reporting validity beside quality rather than folding one into the other." }, { "heading": "The judges are also the judged", "body": "Every model on the panel is also a contender, which is the sharpest threat to these numbers. Label blinding removes names, not stylistic fingerprints. Judge marginal means range from 16.155 to 17.191, a 1.036-point leniency spread that exceeds the 0.55-point first-versus-second headline gap. Within-contender own-model residuals — a narrower slice than the own-family residuals charted below — are small and sign-inconsistent: -0.37 for Claude Opus 5, -1.10 for GPT-5.6 Sol, +0.50 for Claude Fable 5, and -0.43 for Gemini 3.6 Flash. Raw own-family differences confound judge behavior with contender quality, and the fixed family-aligned batches confound family with comparison context. Removing both Anthropic judges still leaves Claude Opus 5 first, but mid-table ranks move. These checks bound the result; they do not calibrate it against humans." }, { "heading": "Rendered frames are evidence, and they must match", "body": "Judges normally receive two rendered frames per contender alongside the JSON. That only means something if the frame depicts the recorded run being scored. An early version of this pipeline rendered whichever run the arena displayed most recently while the panel scored the first valid run — a mismatch on one hundred of one hundred and ten pairs. Capture and judging now derive the run from one shared selection rule, and the captured run id is recorded so a stale frame cannot be silently reused. Frames are also all-or-nothing per batch: if any contender lacks its full set, the whole batch drops to JSON-only, so no model is judged on text while its peers are judged on pictures. That fallback fired for mechanism linkage when both GLM-5.2 frames failed, so the GLM-5.2, Kimi K3, and Nemotron batch used JSON-only evidence for all four judges — twelve of 440 judgment rows." }, { "heading": "What this panel does not control for", "body": "Three of the four judges are invoked with an explicit high reasoning effort. The fourth, Gemini 3.6 Flash, carries its effort in the route name itself and takes no separate effort flag, so the panel is not perfectly matched on that axis. The four judges made 120 batch invocations rather than 440 independent calls, and contender order and family-aligned batch composition were fixed. Their marginal means differ by 1.036 points. Selected-spec character count also correlates positively with panel score in 9 of 10 tasks (mean within-task r=0.38), but the corpus cannot separate genuine scene richness from preference for elaboration. The study has no human calibration set and no repeat judgments. None of these invalidate the descriptive table; all of them sharply limit rank-order claims." } ] }, "findings": [ { "title": "A trivial probe does not prove a route works", "detail": "One provider route answered a one-line health check and then returned \"Model not found\" on every real generation. Reachability checks that send a toy prompt will certify a dead lane. The smoke test that matters sends the actual task and validates the actual contract.", "impact": "high" }, { "title": "Tool access silently changed what was being measured", "detail": "Given a write tool, agentic models wrote the SceneSpec to a file and replied with a prose summary, which scored as malformed. That is a delivery-channel artefact, not a capability gap, and it penalised exactly the models trained hardest to use tools. Running every contender with tools, skills, and rules disabled made the channel uniform.", "impact": "high" }, { "title": "Rendered evidence can drift from the scored artefact", "detail": "Frame capture drove the interactive arena, which shows a model's newest recorded run, while the judge scored the first valid recorded run. The two disagreed on one hundred of one hundred and ten model-task pairs. Nothing errored; the numbers would simply have been wrong.", "impact": "critical" }, { "title": "Blank-frame heuristics reject legitimately dark scenes", "detail": "A lit-pixel ratio threshold discarded valid renders of genuinely dark scenes. Replacing it with a check for a perfectly uniform image — nothing rendered at all — kept the guard against the real failure without punishing art direction.", "impact": "medium" }, { "title": "Specification-following, not syntax, separates the field", "detail": "Invalid submissions rarely fail to parse. They parse and then omit a required field the prompt named explicitly. Models differ far more in how completely they honour a written contract than in whether they can emit well-formed JSON.", "impact": "medium" }, { "title": "Raw family means overstate self-preference", "detail": "Comparing a judge’s mean for its own family against its mean for others conflates bias with contender quality. On the full corpus the raw comparison implies that the Anthropic judges favour Anthropic contenders by +0.9 and +0.7 points. Within-contender own-model residuals instead range from -1.10 to +0.50 and change sign across judges. Removing both Anthropic judges leaves the top model first, but mid-table ranks move. The fixed family-aligned batches still confound family with context, so the correct conclusion is that simple family means are misleading—not that self-preference has been eliminated.", "impact": "high" }, { "title": "A judge-redundancy result reversed between 59 and 110 submissions", "detail": "On a 59-run subset the two same-family judges looked strongly redundant: their residuals correlated at 0.502 against a cross-family control of 0.114, a 4.4x gap that argued for dropping one of them. On the complete 110-run corpus the same measurement gives 0.559 against a cross-family pair at 0.520 — a difference of 0.039, and no basis for removing a judge. The panel was kept at four. Half a corpus was enough to print a provisional table, but not enough to justify changing the instrument.", "impact": "high" } ], "painPoints": [ { "title": "Provider reachability churns mid-sweep", "detail": "Across one sweep a gateway went offline, an OAuth token expired, and a subscription lapsed. Each looked like a model failure in the logs. The runner now classifies transport faults separately from model faults, records them as retryable, and never lets one become a published zero." }, { "title": "SQLite has one writer", "detail": "Running one judge process per panel member is the cheapest parallelism available, but four writers on one database will collide. WAL plus a busy timeout makes it safe; without both, a collision discards a judge call that has already been paid for." }, { "title": "Judging is the cost and latency long pole", "detail": "Generation is embarrassingly parallel and finishes in under an hour. Judging is four panel members times ten tasks times three batches, each a slow multimodal call with images attached. Sequential judging took longer than the entire generation sweep." }, { "title": "No judge is outside the contest", "detail": "Every model strong enough to judge this rubric is also strong enough to be a contender, so the panel cannot be made independent by selection. The honest response is to measure the contamination and publish it, not to claim it away." }, { "title": "Capture is coupled to the application's display semantics", "detail": "Frames come from the real arena, which is what makes them faithful, but it means any change to what the arena chooses to display silently changes the evidence. The recorded run is now pinned explicitly in the URL and stored with the frame." }, { "title": "Two recorded runs per cell is thin", "detail": "Best-of-two rewards a model that gets lucky once and gives a noisy per-recorded-run validity rate. It is enough to produce a complete descriptive table, not enough to establish stable ranks or support narrow confidence intervals." }, { "title": "Effort settings are not uniform across the panel", "detail": "Reasoning effort is expressed two different ways by two different providers: as a flag for three judges and as part of the route name for the fourth. There is no single field that means \"try equally hard\" across vendors, so a panel assembled from several providers cannot be matched on effort by configuration alone." }, { "title": "Half a corpus is enough to mislead a design decision", "detail": "Intermediate results are available long before a sweep finishes and they are tempting to act on. One redundancy measurement moved by a factor of four between the midpoint and the end, in the direction that would have removed a judge for no reason. Partial data is for monitoring progress, not for changing the experiment." } ], "roadmap": [ { "version": "v2", "title": "Statistical power", "items": [ "Raise recorded runs per cell from two to at least five so validity and within-cell generation variance are estimable.", "Report inter-judge reliability and marginal leniency per rubric dimension rather than a single correlation range.", "Model task, run, and judge variation hierarchically; treat tasks as the sampling unit and refuse to print a strict rank order when uncertainty overlaps." ] }, { "version": "v2", "title": "Judge integrity", "items": [ "Recruit at least one capable judge that is not on the contender roster, even if it is weaker.", "Randomise contender order and rebalance batch composition across tasks and judges so family is not confounded with context.", "Publish the leave-one-out residual and leave-one-family-out sensitivity beside every headline score as standing artifacts." ] }, { "version": "v2", "title": "Human grounding", "items": [ "Run a human-rater agreement study on a stratified sample and report calibration by rubric dimension.", "Publish a per-judgment evidence manifest with selected run id, prompt hash, frame fractions and hashes, evidence mode, judge route, and renderer revision.", "Publish high-disagreement and evidence-fallback cases, which are more informative than an aggregate agreement rate." ] }, { "version": "v3", "title": "Task and scoring depth", "items": [ "Expand the task registry with calibrated difficulty tiers so ceiling effects are visible.", "Add deterministic seeded replay so a render can be reproduced byte-for-byte from a spec.", "Score cost and latency alongside quality, since a model that needs six minutes per scene is not interchangeable with one that needs ninety seconds." ] } ], "references": [ { "title": "Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces", "citation": "Jihan Yang et al., “Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces,” arXiv:2412.14171 (2024).", "url": "https://arxiv.org/abs/2412.14171", "kind": "benchmark", "relevance": "VSI-Bench is primarily perception and question answering over observed video. Prism Arena starts from a text task, requires the contender to emit a SceneSpec v1 JSON program, then renders and judges the resulting 3D animation. It therefore tests constructive spatial competence rather than recall alone." }, { "title": "3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark", "citation": "Wufei Ma et al., “3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark,” arXiv:2412.07825 (2024; revised 2025).", "url": "https://arxiv.org/abs/2412.07825", "kind": "benchmark", "relevance": "3DSRBench asks models to infer answers from visual inputs; Prism asks models to construct an executable scene that visibly satisfies spatial, kinematic, material, and temporal constraints. Prism’s output can fail at schema, rendering, composition, or motion even when a model can verbalize the right relation." }, { "title": "IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering", "citation": "Parker Liu et al., “IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering,” arXiv:2506.23329 (2025).", "url": "https://arxiv.org/abs/2506.23329", "kind": "benchmark", "relevance": "This is the closest conceptual neighbor. IR3D-Bench begins with an image and evaluates reconstruction; Prism begins with an authored scene-and-motion brief and compares general-purpose LLMs under one fixed SceneSpec/React Three Fiber/Anime.js harness. Prism also emphasizes temporal choreography and cross-model arena comparison, not inverse rendering accuracy." }, { "title": "T$^3$Bench: Benchmarking Current Progress in Text-to-3D Generation", "citation": "Yuze He et al., “T³Bench: Benchmarking Current Progress in Text-to-3D Generation,” arXiv:2310.02977 (2023).", "url": "https://arxiv.org/abs/2310.02977", "kind": "benchmark", "relevance": "T³Bench chiefly evaluates generated 3D content from NeRF-era text-to-3D systems and highlights multi-object and surroundings failures. Prism evaluates LLM-authored, executable multi-object scenes and animations in a shared renderer; it is about symbolic scene construction and choreography rather than optimizing a continuous 3D representation." }, { "title": "LayoutGPT: Compositional Visual Planning and Generation with Large Language Models", "citation": "Weixi Feng et al., “LayoutGPT: Compositional Visual Planning and Generation with Large Language Models,” arXiv:2305.15393 (2023).", "url": "https://arxiv.org/abs/2305.15393", "kind": "benchmark", "relevance": "LayoutGPT stops largely at layout planning and evaluates a particular method; it is not a general leaderboard for full rendered animation. Prism’s schema controls scene objects, transforms, materials, and animation, and the common runtime turns those plans into directly inspectable artifacts." }, { "title": "LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study", "citation": "Dongil Yang et al., “LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study,” arXiv:2505.19510 (2025).", "url": "https://arxiv.org/abs/2505.19510", "kind": "benchmark", "relevance": "TSG Bench evaluates a symbolic representation without requiring that it execute or render correctly. Prism likewise uses a structured scene representation, but semantic success is tested downstream in a real renderer and extends from static relations to mechanisms, trajectories, stagger, waves, and other temporal behavior." }, { "title": "JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models", "citation": "Saibo Geng et al., “JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models,” arXiv:2501.10868 (2025).", "url": "https://arxiv.org/abs/2501.10868", "kind": "benchmark", "relevance": "JSONSchemaBench asks whether a decoder can produce structurally compliant JSON across many schemas. Prism uses one domain schema and asks the harder downstream question: does valid JSON denote a compelling, instruction-following 3D animation when executed? Schema validity is a prerequisite in Prism, not the final outcome." }, { "title": "Evaluating Large Language Models Trained on Code", "citation": "Mark Chen et al., “Evaluating Large Language Models Trained on Code,” arXiv:2107.03374 (2021).", "url": "https://arxiv.org/abs/2107.03374", "kind": "benchmark", "relevance": "HumanEval gives code a discrete correctness oracle. Prism shares the executable-verification principle—the generated artifact is actually run—but visual quality and many constraint-satisfaction questions are open-ended, so execution and renderability cannot replace perceptual/rubric judgment." }, { "title": "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", "citation": "Carlos E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?,” arXiv:2310.06770 (2023; ICLR 2024).", "url": "https://arxiv.org/abs/2310.06770", "kind": "benchmark", "relevance": "SWE-bench measures maintenance in existing codebases with test-defined acceptance. Prism measures greenfield declarative scene generation in a fixed runtime. Both reward outputs that survive execution, but Prism’s central criteria include spatial composition, material appearance, and motion, which are not fully captured by tests." }, { "title": "SVGenius: Benchmarking LLMs in SVG Understanding, Editing and Generation", "citation": "Siqi Chen et al., “SVGenius: Benchmarking LLMs in SVG Understanding, Editing and Generation,” arXiv:2506.03139 (2025).", "url": "https://arxiv.org/abs/2506.03139", "kind": "benchmark", "relevance": "SVGenius targets mostly static 2D vector code and covers multiple SVG tasks. Prism targets 3D, time-dependent scenes in one deliberately constrained scene language, with a common renderer and animation driver. SVGenius is a useful precedent for treating graphics code as both syntax and rendered semantics." }, { "title": "Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks", "citation": "Adrienne Deganutti et al., “Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks,” arXiv:2604.04192 (2026; author names listed alphabetically).", "url": "https://arxiv.org/abs/2604.04192", "kind": "benchmark", "relevance": "GraphicDesignBench is the strongest direct precedent for structured graphics and animation, but it focuses on layered 2D professional design and mixes many specialized model modalities and quantitative metrics. Prism is narrower and deeper on programmatic 3D mechanisms and choreography, with every contender producing the same SceneSpec consumed by the same R3F/Anime.js runtime and judged on one shared rubric." }, { "title": "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", "citation": "Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” arXiv:2306.05685 (2023), §4.2.", "url": "https://arxiv.org/abs/2306.05685", "kind": "methodology", "relevance": "This supports LLM judging as a scalable approximation, not as a universal substitute for human validation. The result was established on dialogue, not animated 3D artifacts, and close model pairs are precisely where agreement is weakest. Prism needs a stratified human calibration set before it can claim human-aligned visual quality; panel consensus alone cannot establish that." }, { "title": "LLM Evaluators Recognize and Favor Their Own Generations", "citation": "Arjun Panickssery et al., “LLM Evaluators Recognize and Favor Their Own Generations,” arXiv:2404.13076 (2024), §§2.2 and 3.5.", "url": "https://arxiv.org/abs/2404.13076", "kind": "methodology", "relevance": "Prism’s blind contender identifiers remove a straightforward source-label bias, which is good. But the judges may still infer model family from recurring visual composition, naming, object count, or animation style. Blinding should therefore be reported as a mitigation, not as proof that self-preference has been eliminated; metadata, source code, filenames, and deterministic ordering should also be scrubbed." }, { "title": "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge", "citation": "Lianmin Zheng et al., arXiv:2306.05685 (2023); Lin Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge,” arXiv:2406.07791 (2024).", "url": "https://arxiv.org/abs/2406.07791", "kind": "methodology", "relevance": "If Prism uses independent absolute scoring, classic pairwise A/B position bias is less directly applicable and should not be overstated. It still matters wherever multiple candidates, recorded runs, frames, or rubric dimensions are shown together or serially. Candidate and run order should be randomized or counterbalanced, and any future pairwise arena should score both orders." }, { "title": "Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators", "citation": "Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto, “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators,” arXiv:2404.04475 (2024).", "url": "https://arxiv.org/abs/2404.04475", "kind": "methodology", "relevance": "If judges only see rendered frames, textual verbosity bias is mostly screened off. If they see prompts, rationales, or SceneSpec source, longer specifications may gain an unfair advantage. Even image-only judging can have a visual analogue—rewarding detail density or spectacle over task fidelity—so Prism should separate constraint satisfaction from richness rather than assume the 2D/3D analogue is harmless." }, { "title": "Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models", "citation": "Pat Verga et al., “Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models,” arXiv:2404.18796 (2024), §§4.3–4.4.", "url": "https://arxiv.org/abs/2404.18796", "kind": "methodology", "relevance": "A single judging pass with one rubric prompt provides no estimate of prompt variance. Prism should freeze and publish the prompt for reproducibility, but also run prompt paraphrase/repetition sensitivity checks before treating small rank differences as stable. Absolute rubric scores should be calibrated separately for spatial fidelity, visual quality, and motion rather than assumed transferable from general judging ability." } ] }