Zhipu AI's GLM-5.2 listed alongside GLM-5.1 as awaiting a contribution (no
runs yet). Added to both README pending lines.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #9 added models/kimi-k2-7-code.json but not the README snapshot rows; sync
both zh + en tables (Moonshot AI · Kimi Code · Thinking · 01).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-shot, unmodified outputs for Qwen3.7-Max (Alibaba) run in Qoder, default
thinking; contributor Nagi-ovo.
- outputs/qwen3-7-max/mythos-craft.html — Three.js voxel world (chunked
terrain, first-person controls, break/place, fantasy structures).
- outputs/qwen3-7-max/pelican-cycling.svg — hand-authored seaside SVG, pelican
on a bicycle with a side-profile study inset.
Filenames normalized to the canonical case ids and the folder renamed to match
the model id (qwen3-7-max). Moved out of Pending in both README registries.
Validates: Data OK, 25 models.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two one-shot runs generated in the Vibe web app (Mistral's renamed
Le Chat, Thinking enabled), contributed by Nagi-ovo.
mythos-craft is a fixed build (1 line changed): the world synchronously
built the full RENDER_DISTANCE=4 ring (9x9 = 81 chunks) on load, each
chunk doing an O(voxels x structures) linear scan over mythStructures
for all 16x16x128 voxels, so the main thread froze before the first
frame and the page never opened. RENDER_DISTANCE 4 -> 1 (a 3x3 = 9-chunk
first-screen load) lets it open at 60fps; terrain/visuals/gameplay are
the model's original output, otherwise unmodified. pelican-cycling is
unmodified. Both documented in the run notes; READMEs updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cursor's Composer "Fast" is a serving tier (lower latency, higher cost, same
model intelligence) — not a reasoning-effort level. Drop the Fast/effort framing
entirely: effort is Default, notes just state a plain one-shot Composer 2.5 run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Cursor "Fast" run was registered as label "Composer 2.5 Fast" with effort
"Max" — contradictory, and it baked the speed mode into the model name. Per the
model x harness x effort convention (cf. the Grok Build TUI entry), the label is
just "Composer 2.5" and Fast is the effort, so it sits next to the Cursor · Max
entry as a thinking-budget comparison. Fixed the run notes to match, and synced
both README registry tables (which had also missed the Cursor · High Claude
Fable 5 entry from the prior merge).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match the site change — the community leaderboard now unlocks after 3 votes
in any mix of cases, not 3 in each.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A blockquote at the top of the contributing section, addressed to coding
agents: states the unit (Agent = model × harness) and the id convention,
names the two current cases with cases.json as the verbatim-prompt source,
and explains how to find open slots from cases.json + models/*.json (and the
Registry "Runs" column) so it stays correct as contributions land.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A missing effort reads as Max by the repo's "effort defaults to max"
convention; this run was the harness default, so set it explicitly to
"Default" and surface it in both README registry rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lead the model-vs-harness section with the equation Model + Harness =
Agent (English equation + Chinese gloss): a model is just weights; wrapped
in a harness it becomes an agent that does the work, and that agent is the
unit this bench evaluates. Effort is reframed as a parameter of the agent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New Agent composer-2-5-grok-build (Composer 2.5 model in the Grok Build TUI
harness, contributor Nagi-ovo), two runs: mythos-craft + pelican-cycling.
mythos-craft fix (1 line): the WOOD block defines only top+side faces (no
bottom), but the atlas builder requests top/side/bottom for every block.
Fetching WOOD's bottom read t.bottom.image off undefined and threw a
TypeError at the top level of initTextures(), aborting init before the
"enter world" click listener attached — so clicking the start button did
nothing. getFaceImage now falls back to an existing face when one is
missing (bottom→top, the correct ring end-grain for a log). Verified in
headless Chrome: click now fires requestPointerLock and the overlay hides.
Artifacts renamed to the case-id convention; README registry rows synced
(Composer 2.5 now 3 runs across Cursor + Grok Build TUI).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One row per model with its harness x effort combinations listed
together; generated from models/*.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The showcase app and the arena backend now live in the private
nagi-bench-site repo and deploy to Cloudflare Pages, fetching this
repo's models/, outputs/ and cases.json at build time. CI here is a
standalone data validation (same rules as before, no build), plus an
optional repository_dispatch that triggers an instant site rebuild when
SITE_DISPATCH_TOKEN is configured — otherwise the site's 6-hour cron
picks new data up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turnstile tokens are single-use — the old implementation memoized one
token forever, so every batch after the first would fail verification.
Each sync now runs a fresh invisible challenge, and vote syncing is
debounced 2.5s so rapid ballots coalesce into one batch + one challenge.
OG count 16 -> 17 ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every (voter, case, pair) now gets exactly one vote: the client sampler
only draws matchups the browser hasn't voted on (with a graduation state
once all are spent, counter shows voted/total pairs), and the worker
normalizes pair order on insert under a unique (ip, case, pair) index so
side-swapping can't double-count. Daily IP cap drops 300 -> 200 since a
voter graduates at ~171 pairs today.
Matchmaking is now active rather than uniform: unvoted pairs are drawn
with weight closeness/(1+battles) — close ratings and under-played
matchups surface more, unrated combos get placement matches first — with
a 25% pure-exploration floor. Ratings come from the community pool (60s
cached) and fall back to local votes. Rankings stay Bradley-Terry.
Also registers the empty outputs/claude-opus-4-8/ folder as a pending
registry entry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each case now defaults to Arena mode: two anonymous artifacts (identities
hidden, footers suppressed) battle via A / B / Neither, reveal after the
vote, with the same clip-reveal motion language as the rest of the site.
Demo Show replaces the per-combination tab strip with one tab per model;
multi-harness models get a harness switcher and a side-by-side compare,
so the lineup no longer bloats as harnesses multiply.
Votes live in lib/arena.ts (localStorage, custom event fan-out) and rank
via Bradley-Terry fits on pairwise wins, LMSYS Chatbot Arena style, with
'neither' surfaced as a both-bad rate. The #board section stays locked
until 3 votes are cast in each case (blind first, board after), then
shows your votes plus the ranking. When VITE_ARENA_API is configured the
store syncs to the zero-dependency Cloudflare Worker + D1 backend in
worker/ (Turnstile verification, per-IP daily caps, idempotent retries);
deploy steps live in worker/README.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per PR #2's description those two pelicans were generated in Google AI
Studio, not AntiGravity — they move to gemini-3-1-pro-aistudio and
gemini-3-5-flash-aistudio entries (harness Google AI Studio, High).
XA2333's unknown-environment runs stay where they were, disclaimed in
their notes. Adds a bilingual PR template asking for harness/effort/
one-shot info up front, and registers the new Kimi K2.6 mythos-craft
artifact (16 combinations ran).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kimi K2.6 (Kimi Code, local) and MiniMax M3 (MiniMax Code web, Thinking
mode) graduate from pending to ran — 14 combinations ran. Harness icon
rules added for Kimi/MiniMax; READMEs and OG count updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The minecraft.html artifact is Fable 5 Max run inside Claude Code — a
different harness from the existing Claude Web App entry, so per the
model x harness x effort architecture it becomes a new combination
(claude-fable-5-code) rather than a variant. The validator now also
rejects loose files under outputs/<id>/ that no run declares, which is
how this artifact briefly slipped into the repo unregistered. OG count
refreshed (11 -> 12 ran).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There is no "GPT-5.5 Codex" model — Codex is the harness. The entry is
GPT-5.5 run in Codex CLI at xhigh effort, which the harness and effort
fields already record; READMEs and code comments updated to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bump body text across the registry and case sections (notes, prompts,
tabs, table cells) one size up for readability; SVG viewer no longer
uses the full-bleed breakout and caps at a smaller height (HTML keeps
the big frame).
- Brand icons for model, vendor and harness via @lobehub/icons-static-svg
(zero-dep static assets; deliberately not the @lobehub/icons React
package, whose peers pull in antd). Mono SVGs are currentColor so they
follow the theme. Mapping lives in site/src/lib/brands.tsx, derived
from the strings contributors already fill in models/*.json.
- Case pages now show Harness (with icon) and Effort chips next to the
contributor chip; registry harness column is labeled 运行环境（Harness）.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Runs gain a contributor field (GitHub username); the site renders an
avatar chip linked to their profile next to each run's notes, and CI
validates the field. All existing runs are attributed.
- SVG variants of a case are prefetched during idle time, so switching
model tabs swaps images without a network hiccup.
- Xiaomi MiMo v2.5 Pro (Claude Code, max effort) integrated from PR #4
with both case runs and provenance notes; README registry tables
updated.
Co-authored-by: DuJiabao <Jiabao.du@Yunzhisun.com>
Each entry is now explicitly a model x harness x effort combination:
models/*.json gains harness and effort fields, GPT-5.5 xhigh is
relabeled GPT-5.5 Codex (the harness distinguishes it from GPT-5.5
Pro), and the scrolling marquee is replaced by a readable registry
table (model / vendor / harness / effort / runs, pending rows dimmed)
with an explainer on the model-vs-harness relationship.
Top bar controls enlarged: segmented zh/EN switch and bordered 36px
icon buttons. README rewritten Chinese-first with a hand-maintained
registry snapshot table; README.en.md mirrors it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributors no longer touch site code. Model registration moves from
cases.ts literals to repo-root models/<model-id>.json files (filename =
model id = outputs/ folder); cases move to cases.json; cases.ts now just
derives MODELS/RUNS via import.meta.glob. Parallel PRs stop conflicting
because each model owns its own file.
scripts/validate-data.ts runs inside `bun run build` and a new PR CI
workflow: dash-only ids, required bilingual provenance note per run,
declared artifacts must exist, no orphan outputs/ dirs.
Also: run notes render whitespace-pre-wrap (multi-line notes, idea from
PR #2), README contribution guide rewritten around the JSON template,
CLAUDE.md updated, OG card refreshed for the DeepSeek merge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Vite base '/' and root favicon for the custom domain
- Artifact paths normalized to outputs/<model-id>/<case-id>.<ext>
(pelican SVG renamed); RUNS now derives the path by convention,
with an optional file override
- Deep links: #<case-id> anchors a case, #<case-id>:<model-id> also
preselects the model tab; tab clicks replaceState the hash so the
current view is always shareable
- README documents the URL scheme and the legacy github.io 301s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One-shot LLM eval cases with a bilingual (zh/EN) showcase site:
Vite + React + Tailwind v4 + GSAP, built with bun, deployed to
GitHub Pages. Ships two Claude Fable 5 Max cases (Mythos Craft
voxel game, pelican-by-the-sea SVG) with placeholder slots for
GPT-5.1, Gemini 3 Pro, DeepSeek V3.2 and Qwen3 Max.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
