nagi-bench/AGENTS.md

AGENTS.md

Guidance for humans and coding agents contributing to NAGI BENCH data (https://bench.nagi.fun/). Read this before opening a PR.

What this repo is

The public data repo: community-contributed one-shot LLM eval artifacts. You add a model's raw output for a case; the site renders it verbatim. This repo holds data only — no site code, no build of the page you see.

A contribution is exactly two files

1. The artifact — the raw model output, dropped at outputs/<artifactDir>/<case-id>.<ext> (e.g. a single-file .html or .svg).
2. The registration — models/<model-id>.json declaring it.

Do not touch anything else. In particular:

• Do not edit README.md / README.en.md. The Registry tables are generated from models/*.json and synced automatically after merge. Hand edits are overwritten.
• Do not edit cases.json. Cases are maintainer-owned.

models/<model-id>.json schema

{
  "label": "DeepSeek V4 Flash",   // human-facing model name (required)
  "vendor": "DeepSeek",            // who makes the model (required)
  "harness": "Reasonix",           // the tool/agent/UI you ran it in (required)
  "effort": "Max",                 // thinking budget / effort level (required)
  "artifactDir": "deepseek-v4-flash/reasonix-max", // optional; defaults to <model-id>
  "order": 50,                      // optional display sort
  "runs": {
    "skeleton-watch": {
      "note": {                     // REQUIRED, bilingual, zh first, no emojis
        "zh": "在 Reasonix 中以 DeepSeek V4 Flash（Max）一次生成，未修改。",
        "en": "One-shot by DeepSeek V4 Flash (Max effort) in Reasonix, unmodified."
      },
      "contributor": "your-github-username"
    }
  }
}

Naming rules

• The filename IS the entry id — lowercase letters, digits and dashes only (deepseek-v4-flash-reasonix). It is the stable voting key and the default outputs/ folder, so name it after the model, not the harness. Bad: reasonix-reasonix-max. Good: deepseek-v4-flash-reasonix.
• artifactDir may be one or two dash-case segments (deepseek-v4-flash/reasonix-max); set it when several entries share a model folder. Leave it out to default to the id.

One entry per (model × harness × effort)

The same model under a different harness or effort is a separate models/*.json file, not an extra variant inside an existing one. "Codex", "Cursor", "Claude Code", "AntiGravity" name the harness, never the model.

Multiple takes of one case

Ran the same model × harness × effort on a case more than once and want to keep several outputs? Make runs.<case-id> an array — this is the only case where extra artifacts live inside one entry (a different harness/effort is still a new file, per above):

"runs": {
  "pelican-cycling": [
    { "note": { "zh": "…", "en": "…" }, "contributor": "alice" },
    { "file": "pelican-coastal-ride.svg", "note": { "zh": "…", "en": "…" }, "contributor": "bob" }
  ]
}

• The first variant may omit file (defaults to <case-id>.<ext>); every extra variant needs its own distinct, descriptive dash-case file (pelican-coastal-ride.svg). Two variants may not resolve to the same file.
• Never name an artifact index.html — it is served as the folder's default page. Use the case name plus a short suffix (mythos-craft-block-era.html).
• Each variant carries its own note and may set its own contributor, so different people can add takes to the same model entry.

The note is required and bilingual

Every run needs note.zh + note.en documenting provenance: which harness, which effort, one-shot or fixed. If you applied any fix, say where the bug was and what you changed. No emojis anywhere (hard rule). zh first.

Run the bare harness — NO skills, NO plugins

Run the model with the harness's default model and effort, but otherwise stripped down: disable every skill and every plugin — including the ones the harness ships with and would auto-invoke — and add no MCP servers and no custom system prompt. A skill or plugin (a built-in design, gsap, game, or diagram helper, an added toolset, etc.) rewrites the output far more than the model itself does, so a run with any of them active measures the tooling, not the model × harness × effort. It is not comparable to other entries and will not be accepted. Disclosing it in note does not fix this: leaderboard identity is only model × harness × effort, so a modified setup gets no separate identity — it would just be mixed into a row it is not comparable with.

Keep the artifact anonymous — no identity leaks

The output must not reveal which model, vendor, or harness produced it. Strip any <!-- Generated by ... --> header, watermark, "as an AI / as a large language model" phrasing, or signature in visible text or code comments before you submit. Blind Arena battles depend on outputs being anonymous, so a self-identifying artifact invalidates the entry.

React cases — contribute src/ only

Some cases (marked kind: "react" in cases.json) ask for a real multi-file React project instead of a single file. For these your artifact is the src/ tree only — multiple files and nested folders are fine — dropped at outputs/<artifactDir>/src/. The site owns the whole toolchain and builds every React case with one shared scaffold (its own installed React + Vite), so there is no per-project dependency setup:

• Do NOT include package.json, a lockfile, node_modules, index.html, or any build / Vite / TS config. The scaffold supplies all of it.
• Use only libraries the site already ships (React + ReactDOM, plus whatever is in its package.json). The build installs nothing new — you cannot pull in arbitrary npm packages.
• The entry point is src/main.tsx, mounting your app into <div id="root">.
• At build time the src/ is bundled and inlined into one self-contained HTML file that the normal viewer serves verbatim. The anonymity rule above still applies: no model / vendor / harness identity anywhere in the code.

Agent cases — maintainer-run only

Some cases (marked kind: "agent" in cases.json, currently turf-war) ask the model to write a game-playing agent that battles other models' agents, ranked by real wins instead of votes. These cases are not open for artifact PRs, by design:

• Policy files are deliberately kept off the public repo — if submissions were public, later entrants could read earlier policies and prompt their model to counter them, which breaks the one-shot fairness of the case.
• All runs are produced by the maintainer, feeding every model the identical canonical prompt one-shot (no test runs, no iteration, no extra hints). Battles are refereed server-side inside an instrumented sandbox with a deterministic compute budget; the browser only replays action logs.
• The registration in models/<model-id>.json (a runs.<case-id> entry with the usual bilingual provenance note) is the only public trace; there is no outputs/ artifact for these cases.
• Want a model or harness combo included? Open an issue and the maintainer will run it.

Review guidelines

NAGI BENCH stores raw benchmark artifacts. Reviewers, including Codex Review, must not request fixes for gameplay, rendering, animation, JavaScript correctness, UX, visual quality, performance, accessibility, or any other internal behavior of files under outputs/. Broken or low-quality artifacts are valid benchmark data when their provenance is documented.

Only flag:

• visible model, vendor, harness, contributor, or other identity leaks inside outputs/ that would break blind Arena review;
• contribution format or schema issues that violate this file or would fail bun scripts/validate-data.ts.

Do not ask contributors to improve or repair the submitted artifacts themselves. If an artifact bug matters, mention it only as optional context for maintainers, not as required PR feedback.

Before you open the PR

bun scripts/validate-data.ts   # the exact schema check CI runs

This enforces: dash-only ids, required bilingual note, declared artifacts exist and are non-empty, no orphan files under outputs/. Green here = green CI.

You do not need to run update-registry.ts — README is synced for you after merge.

Attach a screenshot

The PR template asks for a screenshot of the generation — your harness/IDE showing the model name + effort while producing this artifact. It is the proof that the declared harness / model / effort are real. Paste it into the PR body (GitHub hosts it); do not commit image files into the repo.

Conventions

• No emojis in any data, note, or UI-bound text.
• outputs/ files are served verbatim in a sandboxed iframe. Keep them self-contained; external resources should be well-known CDNs only.