Jesse ZhangandClaude Fable 5
c335473d93
MiMo v2.5 Pro's one-shot outputs wrote the model name into the browser title, an on-screen "Generated by MiMo v2.5 Pro" caption, a console log (mythos-craft.html) and a watermark text at the bottom of the SVG (pelican-cycling.svg), which would reveal its identity during blind Arena voting. Removed those identity strings; generation logic untouched. Documented the surgical fix in both run notes. MiMo was the only entry with a visible model-name leak. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
NAGI BENCH
中文 | English
One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x effort combinations, one attempt and no retries — a side-by-side record of the runnable artifacts they ship.
Live site: https://bench.nagi.fun/
Models vs harnesses
- Model: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5.
- Harness: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
- The unit of evaluation here is therefore the combination: the same model under a different harness or effort level is a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two records of the same model family). Effort defaults to max; new effort variants of an existing combination can be added later.
Registry
| Model | Vendor | Harness x Effort | Runs |
|---|---|---|---|
| Claude Fable 5 | Anthropic | Claude Web App · Max Claude Code · Max |
04 |
| Claude Opus 4.8 | Anthropic | Claude Code · Max | 02 |
| GPT-5.5 Pro | OpenAI | ChatGPT Web · Extended Pro | 02 |
| GPT-5.5 | OpenAI | Codex CLI · xhigh | 02 |
| Gemini 3.1 Pro | AntiGravity · High Google AI Studio · High |
03 | |
| Gemini 3.5 Flash | AntiGravity · High Google AI Studio · High |
03 | |
| Grok Build | xAI | Grok Build TUI · Max | 02 |
| Composer 2.5 | Cursor | Cursor · Max | 01 |
| DeepSeek-V4-Pro | DeepSeek | Claude Code · Max | 02 |
| Doubao Seed 2.0 Pro | ByteDance | Doubao Web · Pro Mode | 01 |
| Doubao Seed 2.0 Mini | ByteDance | Doubao Web · Fast Mode | 01 |
| MiMo v2.5 Pro | Xiaomi | Claude Code · Max | 02 |
| Kimi K2.6 | Moonshot AI | Kimi Code · Thinking | 02 |
| MiniMax M3 | MiniMax | MiniMax Code Web · Thinking | 01 |
Pending: Mistral Medium 3.5 · GLM-5.1 · Qwen3.7-Max (PRs welcome)
This table is a hand-maintained snapshot; the site's registry section is generated live from
models/*.jsonand is authoritative.
Arena blind battles & community board
- Each case defaults to Arena mode: two anonymous artifacts battle (A / B / Neither) and identities reveal only after you vote; switch to Demo Show to browse everything (tabs are grouped by model — a model with several harnesses can be viewed singly or side by side).
- Casting 03 votes in each case unlocks the community leaderboard, ranked by Bradley-Terry scores (the LMSYS Chatbot Arena approach); "neither" counts separately as both-bad.
- Votes stay in the browser by default; deploying
worker/(Cloudflare Worker + D1 + Turnstile, free tier) upgrades the board to sitewide crowdsourced data — see worker/README.md.
Structure
outputs/<model-id>/<case-id>.<ext> raw model artifacts (HTML / SVG), filename = case id
models/<model-id>.json combination registry: label / vendor / harness / effort / order / run notes
cases.json case definitions (bilingual prompts, maintainer-owned)
scripts/validate-data.ts data validation (CI runs it on every PR)
The site and vote-backend source lives in the private
nagi-bench-siterepo, deployed on Cloudflare Pages; this repo is the site's data source.
Validation
bun scripts/validate-data.ts # the same data validation CI runs
Contributing a run
Contributions are data-only — no code changes needed:
- Artifact:
outputs/<model-id>/<case-id>.<ext>(e.g.outputs/gpt-5-5-pro/pelican-cycling.svg). - Registration:
models/<model-id>.json(new file for a new combination; add torunsfor an existing one) — see the schema example in the Chinese README.
CI-enforced rules: dash-only lowercase ids matching the outputs/ folder name; a required bilingual provenance note per run (which harness, what effort, one-shot or fixed); every declared run must have its artifact; multiple versions of the same combination x case go in an array with distinct file names (e.g. pelican-cycling-2.svg). Set contributor to your GitHub username — the site shows your avatar linked to your profile next to the run. Fill in harness and effort truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via lobe-icons) — contributors never touch icons.
PRs get data validation from CI; merges to main rebuild the site automatically (usually instant, at most 6 hours).