nagi-bench/AGENTS.md

AGENTS.md

Guidance for humans and coding agents contributing to NAGI BENCH data (https://bench.nagi.fun/). Read this before opening a PR. 中文说明见每节。

What this repo is / 这个仓库是什么

The public data repo: community-contributed one-shot LLM eval artifacts. You add a model's raw output for a case; the site renders it verbatim. This repo holds data only — no site code, no build of the page you see.

公开数据仓库：社区贡献的「一次生成」LLM 测评产物。你提交某模型在某个 case 上的 原始输出，站点会原样渲染。这里只有数据，没有站点代码。

A contribution is exactly two files / 一次贡献就两个文件

1. The artifact — the raw model output, dropped at outputs/<artifactDir>/<case-id>.<ext> (e.g. a single-file .html or .svg).
2. The registration — models/<model-id>.json declaring it.

Do not touch anything else. In particular:

• Do not edit README.md / README.en.md. The Registry tables are generated from models/*.json and synced automatically after merge. Hand edits are overwritten. （不要改 README，注册表是自动生成并在合并后自动同步的。）
• Do not edit cases.json. Cases are maintainer-owned.（case 由维护者管理。）

models/<model-id>.json schema

{
  "label": "DeepSeek V4 Flash",   // human-facing model name (required)
  "vendor": "DeepSeek",            // who makes the model (required)
  "harness": "Reasonix",           // the tool/agent/UI you ran it in (required)
  "effort": "Max",                 // thinking budget / effort level (required)
  "artifactDir": "deepseek-v4-flash/reasonix-max", // optional; defaults to <model-id>
  "order": 50,                      // optional display sort
  "runs": {
    "skeleton-watch": {
      "note": {                     // REQUIRED, bilingual, zh first, no emojis
        "zh": "在 Reasonix 中以 DeepSeek V4 Flash（Max）一次生成，未修改。",
        "en": "One-shot by DeepSeek V4 Flash (Max effort) in Reasonix, unmodified."
      },
      "contributor": "your-github-username"
    }
  }
}

Naming rules / 命名规则

• The filename IS the entry id — lowercase letters, digits and dashes only (deepseek-v4-flash-reasonix). It is the stable voting key and the default outputs/ folder, so name it after the model, not the harness. Bad: reasonix-reasonix-max. Good: deepseek-v4-flash-reasonix. （文件名即条目 id，是稳定投票主键——按模型命名，别按 harness 命名。）
• artifactDir may be one or two dash-case segments (deepseek-v4-flash/reasonix-max); set it when several entries share a model folder. Leave it out to default to the id.

One entry per (model × harness × effort) / 一个组合一个条目

The same model under a different harness or effort is a separate models/*.json file, not an extra variant inside an existing one. "Codex", "Cursor", "Claude Code", "AntiGravity" name the harness, never the model. （同一模型换 harness 或 effort = 新建 JSON，不要塞进现有条目。）

The note is required and bilingual / note 必须双语

Every run needs note.zh + note.en documenting provenance: which harness, which effort, one-shot or fixed. If you applied any fix, say where the bug was and what you changed. No emojis anywhere (hard rule). zh first.

Run the harness as it ships / 用 harness 开箱即用的默认形态

Use the harness in its default, out-of-the-box configuration: no extra skills, plugins, MCP servers or custom system prompts beyond what it bundles. These shift the output as much as swapping the harness does, so loading them makes the run no longer comparable to others under the same harness. If you used any, it is a different setup — say so explicitly in note, otherwise it does not belong on the board. （用 harness 的默认形态：不要额外加载 harness 自带以外的 skill / 插件 / MCP / 自定义系统提示——它们对产出的影响不亚于换 harness，会让这条 run 失去与 同一 harness 下其它条目的可比性。确实用了就在 note 里写清楚，否则不该上榜。）

Before you open the PR / 提交前

bun scripts/validate-data.ts   # the exact schema check CI runs

This enforces: dash-only ids, required bilingual note, declared artifacts exist and are non-empty, no orphan files under outputs/. Green here = green CI.

You do not need to run update-registry.ts — README is synced for you after merge.（不需要手动跑 update-registry，README 合并后自动同步。）

Attach a screenshot / 附一张截图

The PR template asks for a screenshot of the generation — your harness/IDE showing the model name + effort while producing this artifact. It is the proof that the declared harness / model / effort are real. Paste it into the PR body (GitHub hosts it); do not commit image files into the repo. （PR 里贴一张生成过程截图作为「确实用了这套 harness + 模型 + effort」的凭证， 直接贴进 PR 正文，不要把图片提交进仓库。）

Conventions / 约定

• No emojis in any data, note, or UI-bound text.
• outputs/ files are served verbatim in a sandboxed iframe. Keep them self-contained; external resources should be well-known CDNs only.