nagi-bench/AGENTS.md

# AGENTS.md

Guidance for humans and coding agents contributing to **NAGI BENCH** data
(https://bench.nagi.fun/). Read this before opening a PR.

## What this repo is

The **public data repo**: community-contributed *one-shot* LLM eval artifacts.
You add a model's raw output for a case; the site renders it verbatim. This repo
holds **data only** — no site code, no build of the page you see.

## A contribution is exactly two files

1. **The artifact** — the raw model output, dropped at
   `outputs/<artifactDir>/<case-id>.<ext>` (e.g. a single-file `.html` or `.svg`).
2. **The registration** — `models/<model-id>.json` declaring it.

Do **not** touch anything else. In particular:

- **Do not edit `README.md` / `README.en.md`.** The Registry tables are
  generated from `models/*.json` and synced automatically after merge. Hand
  edits are overwritten.
- **Do not edit `cases.json`.** Cases are maintainer-owned.

## `models/<model-id>.json` schema

```jsonc
{
  "label": "DeepSeek V4 Flash",   // human-facing model name (required)
  "vendor": "DeepSeek",            // who makes the model (required)
  "harness": "Reasonix",           // the tool/agent/UI you ran it in (required)
  "effort": "Max",                 // thinking budget / effort level (required)
  "artifactDir": "deepseek-v4-flash/reasonix-max", // optional; defaults to <model-id>
  "order": 50,                      // optional display sort
  "runs": {
    "skeleton-watch": {
      "note": {                     // REQUIRED, bilingual, zh first, no emojis
        "zh": "在 Reasonix 中以 DeepSeek V4 Flash（Max）一次生成，未修改。",
        "en": "One-shot by DeepSeek V4 Flash (Max effort) in Reasonix, unmodified."
      },
      "contributor": "your-github-username"
    }
  }
}
```

### Naming rules

- **The filename IS the entry id** — lowercase letters, digits and dashes only
  (`deepseek-v4-flash-reasonix`). It is the **stable voting key** and the default
  `outputs/` folder, so name it after the **model**, not the harness. Bad:
  `reasonix-reasonix-max`. Good: `deepseek-v4-flash-reasonix`.
- `artifactDir` may be one or two dash-case segments
  (`deepseek-v4-flash/reasonix-max`); set it when several entries share a model
  folder. Leave it out to default to the id.

### One entry per (model × harness × effort)

The same model under a different **harness** or **effort** is a **separate**
`models/*.json` file, not an extra variant inside an existing one. "Codex",
"Cursor", "Claude Code", "AntiGravity" name the **harness**, never the model.

### Multiple takes of one case

Ran the **same** model × harness × effort on a case more than once and want to
keep several outputs? Make `runs.<case-id>` an **array** — this is the *only*
case where extra artifacts live inside one entry (a different harness/effort is
still a new file, per above):

```jsonc
"runs": {
  "pelican-cycling": [
    { "note": { "zh": "…", "en": "…" }, "contributor": "alice" },
    { "file": "pelican-coastal-ride.svg", "note": { "zh": "…", "en": "…" }, "contributor": "bob" }
  ]
}
```

- The **first** variant may omit `file` (defaults to `<case-id>.<ext>`); **every
  extra** variant needs its own distinct, descriptive dash-case `file`
  (`pelican-coastal-ride.svg`). Two variants may not resolve to the same file.
- **Never name an artifact `index.html`** — it is served as the folder's default
  page. Use the case name plus a short suffix (`mythos-craft-block-era.html`).
- Each variant carries its own `note` and may set its own `contributor`, so
  different people can add takes to the same model entry.

### The `note` is required and bilingual

Every run needs `note.zh` + `note.en` documenting provenance: which harness,
which effort, one-shot or fixed. If you applied any fix, say where the bug was
and what you changed. **No emojis anywhere** (hard rule). zh first.

### Run the bare harness — NO skills, NO plugins

Run the model with the harness's default model and effort, but otherwise
**stripped down**: **disable every skill and every plugin — including the ones
the harness ships with and would auto-invoke** — and add no MCP servers and no
custom system prompt. A skill or plugin (a built-in `design`, `gsap`, game, or
diagram helper, an added toolset, etc.) rewrites the output far more than the
model itself does, so a run with any of them active measures the tooling, not the
model × harness × effort. It is not comparable to other entries and **will not be
accepted**. Disclosing it in `note` does not fix this: leaderboard identity is
only model × harness × effort, so a modified setup gets no separate identity — it
would just be mixed into a row it is not comparable with.

### Keep the artifact anonymous — no identity leaks

**The output must not reveal which model, vendor, or harness produced it.** Strip
any `<!-- Generated by ... -->` header, watermark, "as an AI / as a large
language model" phrasing, or signature in visible text or code comments before
you submit. Blind Arena battles depend on outputs being anonymous, so a
self-identifying artifact invalidates the entry.

## React cases — contribute `src/` only

Some cases (marked `kind: "react"` in `cases.json`) ask for a real multi-file
React project instead of a single file. For these your artifact is **the `src/`
tree only** — multiple files and nested folders are fine — dropped at
`outputs/<artifactDir>/src/`. The site owns the whole toolchain and builds every
React case with **one shared scaffold** (its own installed React + Vite), so
there is no per-project dependency setup:

- **Do NOT include `package.json`, a lockfile, `node_modules`, `index.html`, or
  any build / Vite / TS config.** The scaffold supplies all of it.
- **Use only libraries the site already ships** (React + ReactDOM, plus whatever
  is in its `package.json`). The build installs nothing new — you cannot pull in
  arbitrary npm packages.
- **The entry point is `src/main.tsx`**, mounting your app into `<div id="root">`.
- At build time the `src/` is bundled and inlined into **one self-contained HTML
  file** that the normal viewer serves verbatim. The anonymity rule above still
  applies: no model / vendor / harness identity anywhere in the code.

## Review guidelines

NAGI BENCH stores raw benchmark artifacts. Reviewers, including Codex Review,
must **not** request fixes for gameplay, rendering, animation, JavaScript
correctness, UX, visual quality, performance, accessibility, or any other
internal behavior of files under `outputs/`. Broken or low-quality artifacts are
valid benchmark data when their provenance is documented.

Only flag:

- visible model, vendor, harness, contributor, or other identity leaks inside
  `outputs/` that would break blind Arena review;
- contribution format or schema issues that violate this file or would fail
  `bun scripts/validate-data.ts`.

Do not ask contributors to improve or repair the submitted artifacts themselves.
If an artifact bug matters, mention it only as optional context for maintainers,
not as required PR feedback.

## Before you open the PR

```bash
bun scripts/validate-data.ts   # the exact schema check CI runs
```

This enforces: dash-only ids, required bilingual `note`, declared artifacts
exist and are non-empty, no orphan files under `outputs/`. Green here = green CI.

You do **not** need to run `update-registry.ts` — README is synced for you after
merge.

### Attach a screenshot

The PR template asks for a **screenshot of the generation** — your harness/IDE
showing the model name + effort while producing this artifact. It is the proof
that the declared `harness` / `model` / `effort` are real. Paste it into the PR
body (GitHub hosts it); **do not commit image files** into the repo.

## Conventions

- No emojis in any data, note, or UI-bound text.
- `outputs/` files are served verbatim in a sandboxed iframe. Keep them
  self-contained; external resources should be well-known CDNs only.