Jesse ZhangandClaude Fable 5
bf23fc4168
Every (voter, case, pair) now gets exactly one vote: the client sampler only draws matchups the browser hasn't voted on (with a graduation state once all are spent, counter shows voted/total pairs), and the worker normalizes pair order on insert under a unique (ip, case, pair) index so side-swapping can't double-count. Daily IP cap drops 300 -> 200 since a voter graduates at ~171 pairs today. Matchmaking is now active rather than uniform: unvoted pairs are drawn with weight closeness/(1+battles) — close ratings and under-played matchups surface more, unrated combos get placement matches first — with a 25% pure-exploration floor. Ratings come from the community pool (60s cached) and fall back to local votes. Rankings stay Bradley-Terry. Also registers the empty outputs/claude-opus-4-8/ folder as a pending registry entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
75 lines
4.7 KiB
Markdown
75 lines
4.7 KiB
Markdown
# NAGI BENCH
|
|
|
|
[中文](./README.md) | English
|
|
|
|
One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x effort combinations, one attempt and no retries — a side-by-side record of the runnable artifacts they ship.
|
|
|
|
**Live site:** https://bench.nagi.fun/
|
|
|
|
## Models vs harnesses
|
|
|
|
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5.
|
|
- **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
|
|
- The unit of evaluation here is therefore the **combination**: the same model under a different harness or effort level is a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two records of the same model family). Effort defaults to max; new effort variants of an existing combination can be added later.
|
|
|
|
## Registry
|
|
|
|
| Model | Vendor | Harness | Effort | Runs |
|
|
|---|---|---|---|---|
|
|
| Claude Fable 5 | Anthropic | Claude Web App | Max | 02 |
|
|
| Claude Fable 5 | Anthropic | Claude Code | Max | 02 |
|
|
| GPT-5.5 Pro | OpenAI | ChatGPT Web | Extended Pro | 02 |
|
|
| GPT-5.5 | OpenAI | Codex CLI (no skills) | xhigh | 02 |
|
|
| Gemini 3.1 Pro | Google | AntiGravity | High | 02 |
|
|
| Gemini 3.1 Pro | Google | Google AI Studio | High | 01 |
|
|
| Gemini 3.5 Flash | Google | AntiGravity | High | 02 |
|
|
| Gemini 3.5 Flash | Google | Google AI Studio | High | 01 |
|
|
| Grok Build | xAI | Grok Build TUI (local) | Max | 02 |
|
|
| Composer 2.5 | Cursor | Cursor | Max | 01 |
|
|
| DeepSeek-V4-Pro | DeepSeek | Claude Code | Max | 02 |
|
|
| Doubao Seed 2.0 Pro | ByteDance | Doubao Web | Pro Mode | 01 |
|
|
| Doubao Seed 2.0 Mini | ByteDance | Doubao Web | Fast Mode | 01 |
|
|
| MiMo v2.5 Pro | Xiaomi | Claude Code | Max | 02 |
|
|
| Kimi K2.6 | Moonshot AI | Kimi Code (local) | Thinking | 02 |
|
|
| MiniMax M3 | MiniMax | MiniMax Code Web | Thinking | 01 |
|
|
|
|
Pending: Claude Opus 4.8 · Mistral Medium 3.5 · GLM-5.1 · Qwen3.7-Max (PRs welcome)
|
|
|
|
> This table is a hand-maintained snapshot; the site's registry section is generated live from `models/*.json` and is authoritative.
|
|
|
|
## Arena blind battles & community board
|
|
|
|
- Each case defaults to **Arena mode**: two anonymous artifacts battle (A / B / Neither) and identities reveal only after you vote; switch to **Demo Show** to browse everything (tabs are grouped by model — a model with several harnesses can be viewed singly or side by side).
|
|
- Casting 03 votes in each case unlocks the **community leaderboard**, ranked by Bradley-Terry scores (the LMSYS Chatbot Arena approach); "neither" counts separately as both-bad.
|
|
- Votes stay in the browser by default; deploying `worker/` (Cloudflare Worker + D1 + Turnstile, free tier) upgrades the board to sitewide crowdsourced data — see [worker/README.md](./worker/README.md).
|
|
|
|
## Structure
|
|
|
|
```
|
|
outputs/<model-id>/<case-id>.<ext> raw model artifacts (HTML / SVG), filename = case id
|
|
models/<model-id>.json combination registry: label / vendor / harness / effort / order / run notes
|
|
site/ showcase app (Vite + React + Tailwind v4 + GSAP, bun)
|
|
site/src/data/cases.json case definitions (bilingual prompts, maintainer-owned)
|
|
.github/workflows/ deploy.yml deploys; ci.yml validates data + builds on PRs
|
|
```
|
|
|
|
## Development
|
|
|
|
```bash
|
|
cd site
|
|
bun install
|
|
bun run dev # syncs outputs/ and serves http://localhost:5173/
|
|
bun run build # data validation + type-check + build (site/dist)
|
|
```
|
|
|
|
## Contributing a run
|
|
|
|
Contributions are data-only — no code changes needed:
|
|
|
|
1. Artifact: `outputs/<model-id>/<case-id>.<ext>` (e.g. `outputs/gpt-5-5-pro/pelican-cycling.svg`).
|
|
2. Registration: `models/<model-id>.json` (new file for a new combination; add to `runs` for an existing one) — see the schema example in the Chinese README.
|
|
|
|
CI-enforced rules: dash-only lowercase ids matching the `outputs/` folder name; a **required bilingual provenance `note`** per run (which harness, what effort, one-shot or fixed); every declared run must have its artifact; multiple versions of the same combination x case go in an array with distinct `file` names (e.g. `pelican-cycling-2.svg`). Set `contributor` to your GitHub username — the site shows your avatar linked to your profile next to the run. Fill in `harness` and `effort` truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via [lobe-icons](https://github.com/lobehub/lobe-icons)) — contributors never touch icons.
|
|
|
|
PRs get data validation + a build from CI; merges to `main` deploy automatically.
|