NAGI BENCH

中文 | English

One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x effort combinations, one attempt and no retries — a side-by-side record of the runnable artifacts they ship.

Live site: https://bench.nagi.fun/

Models vs harnesses

Model + Harness = Agent — a model is just weights; wrap it in a runtime and it becomes an agent that can actually do the work. That agent is the unit this bench evaluates.

Model: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5.
Harness: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
The unit of evaluation here is therefore an Agent: the same model under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family). Effort is a parameter of that agent, defaulting to max.

Registry

Model	Vendor	Harness x Effort	Runs
Claude Fable 5	Anthropic	Claude Web App · Max Claude Code · Max Cursor · High	06
Claude Opus 4.8	Anthropic	Claude Code · Max	02
Claude Opus 4.7	Anthropic	Cursor · Max	02
Claude Opus 4.6	Anthropic	Cursor · Max	01
GPT-5.5 Pro	OpenAI	ChatGPT Web · Extended Pro	02
GPT-5.5	OpenAI	Codex CLI · xhigh	02
Gemini 3.1 Pro	Google	AntiGravity · High Cursor · Default Google AI Studio · High	04
Gemini 3.5 Flash	Google	AntiGravity · High Google AI Studio · High Cursor · Default	04
Grok Build	xAI	Grok Build TUI · Max	02
Composer 2.5	Cursor	Cursor · Max Cursor · Default Grok Build TUI · Default	05
Mistral Medium 3.5	Mistral AI	Vibe · Thinking	02
DeepSeek-V4-Pro	DeepSeek	Claude Code · Max	02
Doubao Seed 2.0 Pro	ByteDance	Doubao Web · Pro Mode	01
Doubao Seed 2.0 Mini	ByteDance	Doubao Web · Fast Mode	01
MiMo v2.5 Pro	Xiaomi	Claude Code · Max	02
Kimi K2.6	Moonshot AI	Kimi Code · Thinking	02
Kimi K2.7-Code	Moonshot AI	Kimi Code · Thinking	02
MiniMax M3	MiniMax	MiniMax Code Web · Thinking	01
Qwen3.7-Max	Alibaba	Qoder · Default	02

Pending: GLM-5.1 · GLM-5.2 (PRs welcome)

This table is a hand-maintained snapshot; the site's registry section is generated live from models/*.json and is authoritative.

Arena blind battles & community board

Each case defaults to Arena mode: two anonymous artifacts battle (A / B / Neither) and identities reveal only after you vote; switch to Demo Show to browse everything (tabs are grouped by model — a model with several harnesses can be viewed singly or side by side).
Casting 03 votes total unlocks the community leaderboard, ranked by Bradley-Terry scores (the LMSYS Chatbot Arena approach); "neither" counts separately as both-bad.
Votes stay in the browser by default; deploying worker/ (Cloudflare Worker + D1 + Turnstile, free tier) upgrades the board to sitewide crowdsourced data — see worker/README.md.

Structure

outputs/<artifact-dir>/<case-id>.<ext>  raw model artifacts (HTML / SVG), filename = case id
models/<agent-id>.json                  agent registry: label / vendor / harness / effort / artifactDir / order / run notes
cases.json                           case definitions (bilingual prompts, maintainer-owned)
scripts/validate-data.ts             data validation (CI runs it on every PR)

The site and vote-backend source lives in the private nagi-bench-site repo, deployed on Cloudflare Pages; this repo is the site's data source.

Validation

bun scripts/validate-data.ts   # the same data validation CI runs

Contributing a run

If you are an AI agent (Claude Code / Codex / Cursor, etc.), this part is for you:

The unit you contribute is one Agent = a model × the harness you run in. The <agent-id> (lowercase letters / digits / dashes) encodes that pairing — e.g. gpt-5-5-pro, claude-fable-5-cc (cc = Claude Code). The same model in a different harness is a different Agent with a different id. The <agent-id> is the vote and leaderboard identity; do not rename it just to reorganize files.

There are two cases right now, defined in cases.json: mythos-craft (HTML, a playable voxel world) and pelican-cycling (SVG, a pelican cycling by the sea). The prompt lives in cases.json — use it verbatim, do not paraphrase.

To find what's missing: read cases.json (all cases) and models/*.json (what each Agent has run) — a <case-id> absent from an Agent's json is an open slot; any row in the Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill both.

Then follow the two-file flow below, write a bilingual note (how the artifact was produced — one-shot or fixed), and run bun scripts/validate-data.ts until it passes before opening a PR.

Contributions are data-only — no code changes needed:

Artifact: preferably outputs/<base-model>/<harness-effort>/<case-id>.<ext> (e.g. outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg).
Registration: models/<agent-id>.json (new file for a new combination; add to runs for an existing one) with artifactDir pointing at the artifact folder — see the schema example in the Chinese README.

CI-enforced rules: dash-only lowercase agent-ids as stable identities; without artifactDir, artifacts default to outputs/<agent-id>/; with artifactDir, use one or two lowercase dash-case path segments, preferably <base-model>/<harness-effort>. A bilingual provenance note is required per run (which harness, what effort, one-shot or fixed); every declared run must have its artifact; multiple versions of the same combination x case go in an array with distinct file names (e.g. pelican-cycling-2.svg). Set contributor to your GitHub username — the site shows your avatar linked to your profile next to the run. Fill in harness and effort truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via lobe-icons) — contributors never touch icons.

PRs get data validation from CI; merges to main rebuild the site automatically (usually instant, at most 6 hours).

7.3 KiB Raw Blame History Unescape Escape