Mirror the new contribution rule as a checklist item so contributors confirm
they ran the harness as it ships.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A `note` disclosure does not separate a modified-harness run on the board:
leaderboard identity is only model × harness × effort, so a skill-boosted run
would still land in an Agent row it is explicitly not comparable with. Drop the
"disclose in note" escape hatch and make it a flat non-acceptance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address fairness concern from #24: the same harness can load completely
different skills / plugins / MCP / custom system prompts, which shifts the
output as much as swapping the harness and breaks Agent comparability.
- README.md / README.en.md: add a bullet to the Model + Harness section
stating the harness is evaluated in its default, out-of-the-box form.
- AGENTS.md: add a "Run the harness as it ships" rule — no extra skills
beyond what the harness bundles; disclose any deviation in `note`.
Enforcement stays honor-system + the PR generation screenshot.
Closes#24
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- AGENTS.md: public contributor/agent guide (two-file rule, model-centric
ids, one-entry-per-harness, bilingual notes, screenshot proof)
- PR template: require a generation proof screenshot; drop the manual
update-registry step; add id-naming and no-README-edit checks
- CI: PRs only run validate-data; pushes to main auto-regenerate and commit
the README registry (contents:write on that job, [skip ci], no loop)
- README/update-registry: reflect auto-synced registry; fix stale 'two cases'
-> three (mythos-craft, pelican-cycling, skeleton-watch); link AGENTS.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-shot skeleton-watch generated via codex exec with model gpt-5.5 at
xhigh effort. The prompt was prefixed with an instruction forbidding the
model from reading or referencing any other file in the repo, so it works
purely from its own knowledge. Unmodified original output; renders clean
(no console errors), registered in the model JSON.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tamed bloom (0.28/0.45/0.9) was still blinding when zoomed in: the
intensity-96 key spotlight's near-white specular on the polished plate
kept blooming into an over-exposed haze over the whole movement. Lowering
threshold/strength was not enough, so disable the bloom (strength 0).
The movement is crisp and the lume still reads via its emissive material.
Done out of concern for photosensitive viewers; note updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
UnrealBloomPass used threshold 0.18 / strength 0.72, so with ACES tone
mapping and several strong lights nearly every highlight and lume surface
bloomed into one over-exposed white blob that washed out the whole dial.
Tame it to strength 0.28 / radius 0.45 / threshold 0.9 so the glow only
haloes the lume markers and the watch is legible again. Documented in the
run note (now a 2-line fix).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clicking Exploded View barely moves anything: animate() does
const t = clock.getElapsedTime() then const dt = clock.getDelta().
getElapsedTime() already consumes the frame delta and resets oldTime,
so the following getDelta() returns ~0 and dt is ~0 every frame, leaving
the explode lerp (and crystal offset) effectively frozen. Gear rotations
use t directly so the movement still runs. Bug documented, not fixed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
createSpiralTube passed Math.max(96, turns * 38) as TubeGeometry's
tubularSegments, yielding fractional counts (5.8 -> 220.4, 6.4 -> 243.2).
TubeGeometry's trailing generateSegment indexes the Frenet-frame arrays
with that count, hits undefined and reads .x, throwing a TypeError that
aborts init and leaves the page stuck on the loading overlay forever.
Round the count to an integer; documented in the run note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reorder the Opus (4.8>4.7>4.6>4.5), MiMo (keep the two variants
adjacent) and GLM (5.2>5.1>5 Turbo) entries so the registry sorts by
version high-to-low instead of the previous scrambled order, adjusting
each model JSON's `order` and regenerating the README registry tables.
Also revise the model-vs-harness explainer: reasoning effort is a
capability dial of the model itself (same weights at high vs low effort
reason like different models), so it belongs to the "model" dimension
alongside the weights rather than being framed as a side parameter of
the agent.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename all output artifacts to their canonical case-id names
(mythos-craft.html / pelican-cycling.svg / skeleton-watch.html, with
-2 suffixes for variants) and drop the now-redundant "file" overrides
from the corresponding models/*.json runs.
Register new Claude Code runs for Opus 4.6 and 4.7 (new model JSONs),
add the GPT-5.5 Pro skeleton-watch run, and regenerate the README
registry tables. Remove stray .DS_Store files from outputs/.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Step 3.7 Flash (StepFun) via Claude Code · High, one-shot. The mythos-craft
Write tool call failed during generation, so the model's output was saved into
the HTML by hand (code unmodified) — documented in the run note. Registry
tables regenerated; validation passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zhipu AI's GLM-5.2 listed alongside GLM-5.1 as awaiting a contribution (no
runs yet). Added to both README pending lines.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #9 added models/kimi-k2-7-code.json but not the README snapshot rows; sync
both zh + en tables (Moonshot AI · Kimi Code · Thinking · 01).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swap pelican-cycling.svg to the red-bicycle one-shot version (with a pelican
side-profile study inset); drop the blue-bike variant so only the red one and
its matching mythos-craft.html remain. Note updated to mention the inset.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One-shot, unmodified outputs for Qwen3.7-Max (Alibaba) run in Qoder, default
thinking; contributor Nagi-ovo.
- outputs/qwen3-7-max/mythos-craft.html — Three.js voxel world (chunked
terrain, first-person controls, break/place, fantasy structures).
- outputs/qwen3-7-max/pelican-cycling.svg — hand-authored seaside SVG, pelican
on a bicycle with a side-profile study inset.
Filenames normalized to the canonical case ids and the folder renamed to match
the model id (qwen3-7-max). Moved out of Pending in both README registries.
Validates: Data OK, 25 models.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two one-shot runs generated in the Vibe web app (Mistral's renamed
Le Chat, Thinking enabled), contributed by Nagi-ovo.
mythos-craft is a fixed build (1 line changed): the world synchronously
built the full RENDER_DISTANCE=4 ring (9x9 = 81 chunks) on load, each
chunk doing an O(voxels x structures) linear scan over mythStructures
for all 16x16x128 voxels, so the main thread froze before the first
frame and the page never opened. RENDER_DISTANCE 4 -> 1 (a 3x3 = 9-chunk
first-screen load) lets it open at 60fps; terrain/visuals/gameplay are
the model's original output, otherwise unmodified. pelican-cycling is
unmodified. Both documented in the run notes; READMEs updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cursor's Composer "Fast" is a serving tier (lower latency, higher cost, same
model intelligence) — not a reasoning-effort level. Drop the Fast/effort framing
entirely: effort is Default, notes just state a plain one-shot Composer 2.5 run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Cursor "Fast" run was registered as label "Composer 2.5 Fast" with effort
"Max" — contradictory, and it baked the speed mode into the model name. Per the
model x harness x effort convention (cf. the Grok Build TUI entry), the label is
just "Composer 2.5" and Fast is the effort, so it sits next to the Cursor · Max
entry as a thinking-budget comparison. Fixed the run notes to match, and synced
both README registry tables (which had also missed the Cursor · High Claude
Fable 5 entry from the prior merge).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match the site change — the community leaderboard now unlocks after 3 votes
in any mix of cases, not 3 in each.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A blockquote at the top of the contributing section, addressed to coding
agents: states the unit (Agent = model × harness) and the id convention,
names the two current cases with cases.json as the verbatim-prompt source,
and explains how to find open slots from cases.json + models/*.json (and the
Registry "Runs" column) so it stays correct as contributions land.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A missing effort reads as Max by the repo's "effort defaults to max"
convention; this run was the harness default, so set it explicitly to
"Default" and surface it in both README registry rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lead the model-vs-harness section with the equation Model + Harness =
Agent (English equation + Chinese gloss): a model is just weights; wrapped
in a harness it becomes an agent that does the work, and that agent is the
unit this bench evaluates. Effort is reframed as a parameter of the agent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New Agent composer-2-5-grok-build (Composer 2.5 model in the Grok Build TUI
harness, contributor Nagi-ovo), two runs: mythos-craft + pelican-cycling.
mythos-craft fix (1 line): the WOOD block defines only top+side faces (no
bottom), but the atlas builder requests top/side/bottom for every block.
Fetching WOOD's bottom read t.bottom.image off undefined and threw a
TypeError at the top level of initTextures(), aborting init before the
"enter world" click listener attached — so clicking the start button did
nothing. getFaceImage now falls back to an existing face when one is
missing (bottom→top, the correct ring end-grain for a log). Verified in
headless Chrome: click now fires requestPointerLock and the overlay hides.
Artifacts renamed to the case-id convention; README registry rows synced
(Composer 2.5 now 3 runs across Cursor + Grok Build TUI).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MiMo v2.5 Pro's one-shot outputs wrote the model name into the browser
title, an on-screen "Generated by MiMo v2.5 Pro" caption, a console log
(mythos-craft.html) and a watermark text at the bottom of the SVG
(pelican-cycling.svg), which would reveal its identity during blind
Arena voting. Removed those identity strings; generation logic untouched.
Documented the surgical fix in both run notes. MiMo was the only entry
with a visible model-name leak.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One row per model with its harness x effort combinations listed
together; generated from models/*.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
claude-fable-5-code -> claude-fable-5-cc, claude-opus-4-8 ->
claude-opus-4-8-cc (the entry id doubles as the outputs/ folder name).
The Opus mythos-craft artifact is a newer build of the same one-shot
provenance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The showcase app and the arena backend now live in the private
nagi-bench-site repo and deploy to Cloudflare Pages, fetching this
repo's models/, outputs/ and cases.json at build time. CI here is a
standalone data validation (same rules as before, no build), plus an
optional repository_dispatch that triggers an instant site rebuild when
SITE_DISPATCH_TOKEN is configured — otherwise the site's 6-hour cron
picks new data up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cases.json is data (bilingual case definitions, maintainer-owned), so it
moves alongside models/ and outputs/; the site build will consume it
from here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turnstile tokens are single-use — the old implementation memoized one
token forever, so every batch after the first would fail verification.
Each sync now runs a fresh invisible challenge, and vote syncing is
debounced 2.5s so rapid ballots coalesce into one batch + one challenge.
OG count 16 -> 17 ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The artifact landed unregistered which would fail CI's loose-file check;
note marks harness/effort as to-be-confirmed. worker/.wrangler/ cache is
local tooling and leaves the repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every (voter, case, pair) now gets exactly one vote: the client sampler
only draws matchups the browser hasn't voted on (with a graduation state
once all are spent, counter shows voted/total pairs), and the worker
normalizes pair order on insert under a unique (ip, case, pair) index so
side-swapping can't double-count. Daily IP cap drops 300 -> 200 since a
voter graduates at ~171 pairs today.
Matchmaking is now active rather than uniform: unvoted pairs are drawn
with weight closeness/(1+battles) — close ratings and under-played
matchups surface more, unrated combos get placement matches first — with
a 25% pure-exploration floor. Ratings come from the community pool (60s
cached) and fall back to local votes. Rankings stay Bradley-Terry.
Also registers the empty outputs/claude-opus-4-8/ folder as a pending
registry entry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each case now defaults to Arena mode: two anonymous artifacts (identities
hidden, footers suppressed) battle via A / B / Neither, reveal after the
vote, with the same clip-reveal motion language as the rest of the site.
Demo Show replaces the per-combination tab strip with one tab per model;
multi-harness models get a harness switcher and a side-by-side compare,
so the lineup no longer bloats as harnesses multiply.
Votes live in lib/arena.ts (localStorage, custom event fan-out) and rank
via Bradley-Terry fits on pairwise wins, LMSYS Chatbot Arena style, with
'neither' surfaced as a both-bad rate. The #board section stays locked
until 3 votes are cast in each case (blind first, board after), then
shows your votes plus the ranking. When VITE_ARENA_API is configured the
store syncs to the zero-dependency Cloudflare Worker + D1 backend in
worker/ (Turnstile verification, per-IP daily caps, idempotent retries);
deploy steps live in worker/README.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per PR #2's description those two pelicans were generated in Google AI
Studio, not AntiGravity — they move to gemini-3-1-pro-aistudio and
gemini-3-5-flash-aistudio entries (harness Google AI Studio, High).
XA2333's unknown-environment runs stay where they were, disclaimed in
their notes. Adds a bilingual PR template asking for harness/effort/
one-shot info up front, and registers the new Kimi K2.6 mythos-craft
artifact (16 combinations ran).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kimi K2.6 (Kimi Code, local) and MiniMax M3 (MiniMax Code web, Thinking
mode) graduate from pending to ran — 14 combinations ran. Harness icon
rules added for Kimi/MiniMax; READMEs and OG count updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The minecraft.html artifact is Fable 5 Max run inside Claude Code — a
different harness from the existing Claude Web App entry, so per the
model x harness x effort architecture it becomes a new combination
(claude-fable-5-code) rather than a variant. The validator now also
rejects loose files under outputs/<id>/ that no run declares, which is
how this artifact briefly slipped into the repo unregistered. OG count
refreshed (11 -> 12 ran).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There is no "GPT-5.5 Codex" model — Codex is the harness. The entry is
GPT-5.5 run in Codex CLI at xhigh effort, which the harness and effort
fields already record; READMEs and code comments updated to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bump body text across the registry and case sections (notes, prompts,
tabs, table cells) one size up for readability; SVG viewer no longer
uses the full-bleed breakout and caps at a smaller height (HTML keeps
the big frame).
- Brand icons for model, vendor and harness via @lobehub/icons-static-svg
(zero-dep static assets; deliberately not the @lobehub/icons React
package, whose peers pull in antd). Mono SVGs are currentColor so they
follow the theme. Mapping lives in site/src/lib/brands.tsx, derived
from the strings contributors already fill in models/*.json.
- Case pages now show Harness (with icon) and Effort chips next to the
contributor chip; registry harness column is labeled 运行环境（Harness）.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Runs gain a contributor field (GitHub username); the site renders an
avatar chip linked to their profile next to each run's notes, and CI
validates the field. All existing runs are attributed.
- SVG variants of a case are prefetched during idle time, so switching
model tabs swaps images without a network hiccup.
- Xiaomi MiMo v2.5 Pro (Claude Code, max effort) integrated from PR #4
with both case runs and provenance notes; README registry tables
updated.
Co-authored-by: DuJiabao <Jiabao.du@Yunzhisun.com>
Each entry is now explicitly a model x harness x effort combination:
models/*.json gains harness and effort fields, GPT-5.5 xhigh is
relabeled GPT-5.5 Codex (the harness distinguishes it from GPT-5.5
Pro), and the scrolling marquee is replaced by a readable registry
table (model / vendor / harness / effort / runs, pending rows dimmed)
with an explainer on the model-vs-harness relationship.
Top bar controls enlarged: segmented zh/EN switch and bordered 36px
icon buttons. README rewritten Chinese-first with a hand-maintained
registry snapshot table; README.en.md mirrors it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A combination can now ship multiple versions of the same case:
runs.<case-id> accepts an array, each extra variant naming its own
file; the viewer grows a RUN 01/02 switcher and validation enforces
distinct files plus a provenance note per variant.
Integrated content, attributed to their authors:
- Gemini 3.1 Pro + 3.5 Flash pelican runs from both PRs, shown as two
variants each (PR #2's Google AI Studio runs first, PR #3's second)
- Doubao Seed 2.0 Pro / Mini pelican runs (folders renamed to the
dash-only convention)
- GPT-5.5 Codex pelican run
Co-authored-by: lpluma <liuyz2023@mails.jlu.edu.cn>
Co-authored-by: XA2333 <97206278+XA2333@users.noreply.github.com>
Contributors no longer touch site code. Model registration moves from
cases.ts literals to repo-root models/<model-id>.json files (filename =
model id = outputs/ folder); cases move to cases.json; cases.ts now just
derives MODELS/RUNS via import.meta.glob. Parallel PRs stop conflicting
because each model owns its own file.
scripts/validate-data.ts runs inside `bun run build` and a new PR CI
workflow: dash-only ids, required bilingual provenance note per run,
declared artifacts must exist, no orphan outputs/ dirs.
Also: run notes render whitespace-pre-wrap (multi-line notes, idea from
PR #2), README contribution guide rewritten around the JSON template,
CLAUDE.md updated, OG card refreshed for the DeepSeek merge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Site:
- Cursor follower fades out over iframes (it froze on top of running
artifacts since iframes swallow pointer events) and when the pointer
leaves the window
- HTML viewer: full-bleed width on xl screens (up to 90rem), height
capped to 100dvh-10rem so the frame plus its action row always fits
the viewport (big iframes ate wheel events and trapped page scroll),
fullscreen button (overlay + footer action), and Run now auto-aligns
the frame below the fixed header
Artifacts:
- grok-build: W/S were inverted against the forward vector (W walked
backward); model label renamed to Grok Build
- composer-2-5: added G-key cycle teleport to its three generated
landmarks (elven tower, dragon temple, sky ruins) with a start-screen
hint - they were previously only reachable on foot; ground landmarks
arrive 10 blocks out facing the structure, sky ruins land on the deck
Run notes updated accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- gemini-3-5-flash: ATLAS_COORDS and the canvas tiles share a top-origin
row convention, but default flipY=true mirrored sampling (row r read
row 3-r): grass tops became purple crystal, sides hit the alpha-holed
portal/leaves tiles (the "transparent blocks"). flipY=false, 1 line.
- gpt-5-5-xhigh: movement Z component sign-flipped against the -Z
camera convention, mirroring all motion across the X axis (A/D
swapped facing +-X). Negated Z, 1 line.
Run notes now carry full provenance: GPT-5.5 Pro Extended Pro, Codex
CLI (no skills) for xhigh, AntiGravity high for both Geminis, and the
local Grok Build TUI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Surgical fixes (originals preserved in history), each documented in the
per-model run notes as: where the bug is, pre-fix behavior, root cause,
with the shown build being the fixed one:
- grok-build: 0xAETHER invalid hex literal killed the script at parse
time (pure black screen); atlas flipY mismatch made near terrain
sample undrawn black texels. 2 lines.
- composer-2-5: horizontal movement multiplied dt twice, walking at
~1/60 design speed and making biomes effectively unreachable. 1 line.
- gpt-5-5-pro: 8x2 atlas rows swapped by default flipY, grass became
obsidian and leaves/sand sampled undrawn slots. 1 line.
Gemini 3.5 Flash added as-is (color/transparency analysis next).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Folders renamed to the outputs/<model-id>/<case-id>.html convention
(grok-build per request); model ids normalized to dash-only so they
double as folder names. Gemini 3.1 Pro, Grok and Composer 2.5 flip to
ran status with tabs wired up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Vite base '/' and root favicon for the custom domain
- Artifact paths normalized to outputs/<model-id>/<case-id>.<ext>
(pelican SVG renamed); RUNS now derives the path by convention,
with an optional file override
- Deep links: #<case-id> anchors a case, #<case-id>:<model-id> also
preselects the model tab; tab clicks replaceState the hash so the
current view is always shareable
- README documents the URL scheme and the legacy github.io 301s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One-shot LLM eval cases with a bilingual (zh/EN) showcase site:
Vite + React + Tailwind v4 + GSAP, built with bun, deployed to
GitHub Pages. Ships two Claude Fable 5 Max cases (Mythos Craft
voxel game, pelican-by-the-sea SVG) with placeholder slots for
GPT-5.1, Gemini 3 Pro, DeepSeek V3.2 and Qwen3 Max.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Jesse Zhang