nagi-bench/AGENTS.md

AGENTS.md

Guidance for humans and coding agents contributing to NAGI BENCH data (https://bench.nagi.fun/). Read this before opening a PR. 中文说明见每节。

What this repo is / 这个仓库是什么

The public data repo: community-contributed one-shot LLM eval artifacts. You add a model's raw output for a case; the site renders it verbatim. This repo holds data only — no site code, no build of the page you see.

公开数据仓库：社区贡献的「一次生成」LLM 测评产物。你提交某模型在某个 case 上的 原始输出，站点会原样渲染。这里只有数据，没有站点代码。

A contribution is exactly two files / 一次贡献就两个文件

1. The artifact — the raw model output, dropped at outputs/<artifactDir>/<case-id>.<ext> (e.g. a single-file .html or .svg).
2. The registration — models/<model-id>.json declaring it.

Do not touch anything else. In particular:

• Do not edit README.md / README.en.md. The Registry tables are generated from models/*.json and synced automatically after merge. Hand edits are overwritten. （不要改 README，注册表是自动生成并在合并后自动同步的。）
• Do not edit cases.json. Cases are maintainer-owned.（case 由维护者管理。）

models/<model-id>.json schema

{
  "label": "DeepSeek V4 Flash",   // human-facing model name (required)
  "vendor": "DeepSeek",            // who makes the model (required)
  "harness": "Reasonix",           // the tool/agent/UI you ran it in (required)
  "effort": "Max",                 // thinking budget / effort level (required)
  "artifactDir": "deepseek-v4-flash/reasonix-max", // optional; defaults to <model-id>
  "order": 50,                      // optional display sort
  "runs": {
    "skeleton-watch": {
      "note": {                     // REQUIRED, bilingual, zh first, no emojis
        "zh": "在 Reasonix 中以 DeepSeek V4 Flash（Max）一次生成，未修改。",
        "en": "One-shot by DeepSeek V4 Flash (Max effort) in Reasonix, unmodified."
      },
      "contributor": "your-github-username"
    }
  }
}

Naming rules / 命名规则

• The filename IS the entry id — lowercase letters, digits and dashes only (deepseek-v4-flash-reasonix). It is the stable voting key and the default outputs/ folder, so name it after the model, not the harness. Bad: reasonix-reasonix-max. Good: deepseek-v4-flash-reasonix. （文件名即条目 id，是稳定投票主键——按模型命名，别按 harness 命名。）
• artifactDir may be one or two dash-case segments (deepseek-v4-flash/reasonix-max); set it when several entries share a model folder. Leave it out to default to the id.

One entry per (model × harness × effort) / 一个组合一个条目

The same model under a different harness or effort is a separate models/*.json file, not an extra variant inside an existing one. "Codex", "Cursor", "Claude Code", "AntiGravity" name the harness, never the model. （同一模型换 harness 或 effort = 新建 JSON，不要塞进现有条目。）

The note is required and bilingual / note 必须双语

Every run needs note.zh + note.en documenting provenance: which harness, which effort, one-shot or fixed. If you applied any fix, say where the bug was and what you changed. No emojis anywhere (hard rule). zh first.

Run the harness as it ships / 用 harness 开箱即用的默认形态

Use the harness in its default, out-of-the-box configuration: no extra skills, plugins, MCP servers or custom system prompts beyond what it bundles. These shift the output as much as swapping the harness does, so such a run is not comparable to others under the same harness and will not be accepted. Disclosing it in note does not fix this: the leaderboard identity is only model × harness × effort, so a note gives the modified setup no separate identity — it would still be mixed into an Agent row it is not comparable with. （用 harness 的默认形态：不要额外加载 harness 自带以外的 skill / 插件 / MCP / 自定义系统提示——它们对产出的影响不亚于换 harness，这类 run 与同一 harness 下其它条目不可比，不予收录。在 note 里写明也没用：榜单身份只看 模型 × harness × effort，note 不另立身份，它仍会混进本不该可比的那一行。）

Before you open the PR / 提交前

bun scripts/validate-data.ts   # the exact schema check CI runs

This enforces: dash-only ids, required bilingual note, declared artifacts exist and are non-empty, no orphan files under outputs/. Green here = green CI.

You do not need to run update-registry.ts — README is synced for you after merge.（不需要手动跑 update-registry，README 合并后自动同步。）

Attach a screenshot / 附一张截图

The PR template asks for a screenshot of the generation — your harness/IDE showing the model name + effort while producing this artifact. It is the proof that the declared harness / model / effort are real. Paste it into the PR body (GitHub hosts it); do not commit image files into the repo. （PR 里贴一张生成过程截图作为「确实用了这套 harness + 模型 + effort」的凭证， 直接贴进 PR 正文，不要把图片提交进仓库。）

Conventions / 约定

• No emojis in any data, note, or UI-bound text.
• outputs/ files are served verbatim in a sandboxed iframe. Keep them self-contained; external resources should be well-known CDNs only.