nagi-bench/README.en.md

NAGI BENCH

中文 | English

One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x effort combinations, one attempt and no retries — a side-by-side record of the runnable artifacts they ship.

Live site: https://bench.nagi.fun/

Models vs harnesses

Model + Harness = Agent — a model is just weights; wrap it in a runtime and it becomes an agent that can actually do the work. That agent is the unit this bench evaluates.

• Model: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination.
• Harness: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
• The unit of evaluation here is therefore an Agent: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family).

Registry

This table is generated from models/*.json by bun scripts/update-registry.ts; CI checks that it stays current.

Model	Vendor	Harness x Effort	Runs
Claude Fable 5	Anthropic	Claude Web App · Max Claude Code · Max Cursor · High	06
Claude Opus 4.8	Anthropic	Claude Code · Max	05
Claude Opus 4.7	Anthropic	Claude Code · Max Cursor · Max	05
Claude Opus 4.6	Anthropic	Claude Code · Max Cursor · Max	04
Claude Opus 4.5	Anthropic	Cursor · Thinking	02
Claude Sonnet 4.6	Anthropic	Claude Code · Max Cursor · High	03
Claude Haiku 4.5	Anthropic	Claude Code · Default	02
GPT-5.5 Pro	OpenAI	ChatGPT Web · Extended Pro	03
GPT-5.5	OpenAI	Codex CLI · xhigh	02
GPT-5.4	OpenAI	Cursor · xhigh Codex CLI · xhigh	05
GPT-5.3 Codex	OpenAI	Cursor · xhigh	02
GPT-5.2	OpenAI	Cursor · xhigh	02
Gemini 3.1 Pro	Google	AntiGravity · High Cursor · Default Google AI Studio · High	05
Gemini 3.5 Flash	Google	AntiGravity · High Cursor · Default Google AI Studio · High	05
Grok Build	xAI	Grok Build TUI · Max	02
Nex-N2-Pro	Nex-AGI	Claude Code · Max	02
Grok 4.3	xAI	Cursor · Default	02
Composer 2.5	Cursor	Cursor · Max Cursor · Default Grok Build TUI · Default	05
Mistral Medium 3.5	Mistral AI	Vibe · Thinking	02
DeepSeek-V4-Pro	DeepSeek	Claude Code · Max	02
Doubao Seed 2.0 Pro	ByteDance	Doubao Web · Pro Mode	01
Doubao Seed 2.0 Mini	ByteDance	Doubao Web · Fast Mode	01
MiMo v2.5 Pro	Xiaomi	Claude Code · Max	02
MiMo v2.5 Pro UltraSpeed	Xiaomi	Web · Default	04
Kimi K2.7-Code	Moonshot AI	Kimi Code · Thinking	02
Kimi K2.6	Moonshot AI	Kimi Code · Thinking	02
MiniMax M3	MiniMax	MiniMax Code Web · Thinking	02
MiniMax M2.7	MiniMax	MiniMax Code Web · Thinking	01
GLM-5.2	Zhipu AI	ZCode · Max	03
GLM-5 Turbo	Zhipu AI	ZCode · Thinking	03
Qwen3.7-Max	Alibaba	Qoder · Default	02
Step 3.7 Flash	StepFun	Claude Code · High	02

Pending: GLM-5.1 (PRs welcome)

Arena blind battles & community board

• Each case defaults to Arena mode: two anonymous artifacts battle (A / B / Neither) and identities reveal only after you vote; switch to Demo Show to browse everything (tabs are grouped by model — a model with several harnesses can be viewed singly or side by side).
• Casting 03 votes total unlocks the community leaderboard, ranked by Bradley-Terry scores (the LMSYS Chatbot Arena approach); "neither" counts separately as both-bad.
• Votes stay in the browser by default; deploying worker/ (Cloudflare Worker + D1 + Turnstile, free tier) upgrades the board to sitewide crowdsourced data — see worker/README.md.

Structure

outputs/<artifact-dir>/<case-id>.<ext>  raw model artifacts (HTML / SVG), filename = case id
models/<agent-id>.json                  agent registry: label / vendor / harness / effort / artifactDir / order / run notes
cases.json                           case definitions (bilingual prompts, maintainer-owned)
scripts/validate-data.ts             data validation (CI runs it on every PR)
scripts/update-registry.ts           README Registry table generation (CI checks freshness)

The site and vote-backend source lives in the private nagi-bench-site repo, deployed on Cloudflare Pages; this repo is the site's data source.

Validation

bun scripts/update-registry.ts # updates README Registry tables from models/*.json
bun scripts/validate-data.ts   # the same data validation CI runs

Contributing a run

If you are an AI agent (Claude Code / Codex / Cursor, etc.), this part is for you:

• The unit you contribute is one Agent = a model × the harness you run in. The <agent-id> (lowercase letters / digits / dashes) encodes that pairing — e.g. gpt-5-5-pro, claude-fable-5-cc (cc = Claude Code). The same model in a different harness is a different Agent with a different id. The <agent-id> is the vote and leaderboard identity; do not rename it just to reorganize files.
• There are two cases right now, defined in cases.json: mythos-craft (HTML, a playable voxel world) and pelican-cycling (SVG, a pelican cycling by the sea). The prompt lives in cases.json — use it verbatim, do not paraphrase.
• To find what's missing: read cases.json (all cases) and models/*.json (what each Agent has run) — a <case-id> absent from an Agent's json is an open slot; any row in the script-generated, CI-checked Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill both.
• Then follow the two-file flow below, write a bilingual note (how the artifact was produced — one-shot or fixed), and run bun scripts/validate-data.ts until it passes before opening a PR.

Contributions are data-only — no code changes needed:

1. Artifact: preferably outputs/<base-model>/<harness-effort>/<case-id>.<ext> (e.g. outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg).
2. Registration: models/<agent-id>.json (new file for a new combination; add to runs for an existing one) with artifactDir pointing at the artifact folder — see the schema example in the Chinese README.

CI-enforced rules: dash-only lowercase agent-ids as stable identities; without artifactDir, artifacts default to outputs/<agent-id>/; with artifactDir, use one or two lowercase dash-case path segments, preferably <base-model>/<harness-effort>. A bilingual provenance note is required per run (which harness, what effort, one-shot or fixed); every declared run must have its artifact; multiple versions of the same combination x case go in an array with distinct file names (e.g. pelican-cycling-2.svg). Set contributor to your GitHub username — the site shows your avatar linked to your profile next to the run. Fill in harness and effort truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via lobe-icons) — contributors never touch icons.

PRs get data validation from CI; merges to main rebuild the site automatically (usually instant, at most 6 hours).