Add AGENTS.md + tighten contribution flow

- AGENTS.md: public contributor/agent guide (two-file rule, model-centric
  ids, one-entry-per-harness, bilingual notes, screenshot proof)
- PR template: require a generation proof screenshot; drop the manual
  update-registry step; add id-naming and no-README-edit checks
- CI: PRs only run validate-data; pushes to main auto-regenerate and commit
  the README registry (contents:write on that job, [skip ci], no loop)
- README/update-registry: reflect auto-synced registry; fix stale 'two cases'
  -> three (mythos-craft, pelican-cycling, skeleton-watch); link AGENTS.md

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jesse ZhangandClaude Opus 4.8 committed 2026-06-21 11:29:23 +01:00
1 parent 2fb8499fa8
commit 30983f0e7d
6 files changed
+167 -26

No files matched your search

+24 -10
View File
@@ -1,19 +1,33 @@
<!-- 感谢贡献!只需两类文件:outputs/<model-id>/<case-id>.<ext> + models/<model-id>.json -->
<!-- Thanks for contributing! Only two kinds of files: the artifact + the JSON registration. -->
<!-- 感谢贡献!只需两类文件:outputs/<artifactDir>/<case-id>.<ext> + models/<model-id>.json -->
<!-- Thanks for contributing! Only two kinds of files: the artifact + the JSON registration. 详见 AGENTS.md -->
## 产出说明 / Run info
- 模型 / Model:
- 运行环境 / Harness(如 ChatGPT Web、Claude Code、Google AI Studio、本地 CLI…):
- 思考配额 / Thinking effort(如 Max、High、Extended Pro、Thinking…):
- 是否一次生成 / One-shot? (是/否;若有修复请在 note 里说明 / if fixed, document it in the note)
- 厂商 / Vendor:
- 运行环境 / Harness(如 ChatGPT Web、Claude Code、Codex CLI、AntiGravity…,这是工具不是模型):
- 思考配额 / Thinking effort(如 Max、High、xhigh、Extended Pro、Thinking…):
- 是否一次生成 / One-shot?(是/否;若有修复请在 note 里说明 / if fixed, document it in the note)
## 生成凭证截图 / Proof screenshot **(required)**
<!-- 直接把图片拖进下面,GitHub 会托管它。不要把图片文件提交进仓库。 -->
<!-- Drag the image in below; GitHub hosts it. Do NOT commit image files into the repo. -->
请贴一张**生成过程截图**:你的 harness / IDE 中能看到**模型名 + 思考配额**正在产出本产物,
作为「确实用了这套 harness + 模型 + effort」的凭证。
Please attach a screenshot of the generation showing the **model name + effort** in your harness/IDE.
## 自查 / Checklist
- [ ] `models/<model-id>.json` 中 `harness` 与 `effort` 已如实填写(站点据此生成 metadata 与品牌 icon)
- [ ] 每条 run 的 `note` 双语(zh + en)填写了上面的来历信息 / bilingual `note` documents the provenance above
- [ ] 只改了两类文件:`outputs/<artifactDir>/<case-id>.<ext>` + `models/<model-id>.json` / only the artifact + its JSON
- [ ] **没有**手改 `README.md` / `README.en.md`(注册表合并后由 CI 自动同步)/ did **not** hand-edit the README registry (auto-synced after merge)
- [ ] 模型 id 用短横线、以**模型**命名(非 harness),如 `deepseek-v4-flash-reasonix` / dash-case id named after the model, not the harness
- [ ] `harness` 与 `effort` 如实填写(站点据此生成 metadata 与品牌 icon)/ `harness` and `effort` are accurate
- [ ] 每条 run 的 `note` 双语(zh + en,zh 先,无 emoji)填了来历 / bilingual `note`, zh first, no emojis
- [ ] 同一模型换 Harness 或思考配额是**新建一个 JSON 条目** / a different harness or effort = a NEW json entry
- [ ] `contributor` 是我的 GitHub 用户名 / `contributor` is my GitHub username
- [ ] 同一模型在不同 Harness 或思考配额下是**新建一个 JSON 条目**,不是塞进现有条目 / a different harness or effort = a NEW json entry, not a variant
- [ ] 已运行 `bun scripts/update-registry.ts` 更新 README Registry 表 / ran `bun scripts/update-registry.ts` to refresh the README Registry tables
- [ ] 已贴生成凭证截图 / attached the proof screenshot above
- [ ] 本地 `bun scripts/validate-data.ts` 通过 / `bun scripts/validate-data.ts` passes locally
<!-- CI 会跑数据校验 + 构建;规则详见 README「贡献一个模型产出」。 -->
<!-- 规则详见 AGENTS.md。CI 只跑数据校验;README Registry 在合并到 main 后由 CI 自动重生成。 -->
+26 -2
View File
@@ -14,11 +14,35 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@v2
- name: Check README registry
run: bun scripts/update-registry.ts --check
- name: Validate contribution data
run: bun scripts/validate-data.ts
sync-registry:
# The README Registry tables are generated from models/*.json. Contributors
# do not edit them by hand; on every push to main we regenerate and commit
# any drift back. Commits pushed with GITHUB_TOKEN do not retrigger
# workflows, and the message carries [skip ci] too, so there is no loop.
if: github.event_name == 'push'
needs: validate
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@v2
- name: Regenerate README registry
run: bun scripts/update-registry.ts
- name: Commit if registry drifted
run: |
if git diff --quiet -- README.md README.en.md; then
echo "Registry already in sync."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git commit -am "chore: sync README registry [skip ci]"
git push origin HEAD:main
trigger-site-rebuild:
# The site lives in the private nagi-bench-site repo (Cloudflare Pages).
# With SITE_DISPATCH_TOKEN set the rebuild is instant; without it the
+99
View File
@@ -0,0 +1,99 @@
# AGENTS.md
Guidance for humans and coding agents contributing to **NAGI BENCH** data
(https://bench.nagi.fun/). Read this before opening a PR. 中文说明见每节。
## What this repo is / 这个仓库是什么
The **public data repo**: community-contributed *one-shot* LLM eval artifacts.
You add a model's raw output for a case; the site renders it verbatim. This repo
holds **data only** — no site code, no build of the page you see.
公开**数据仓库**:社区贡献的「一次生成」LLM 测评产物。你提交某模型在某个 case 上的
原始输出,站点会原样渲染。这里只有**数据**,没有站点代码。
## A contribution is exactly two files / 一次贡献就两个文件
1. **The artifact** — the raw model output, dropped at
`outputs/<artifactDir>/<case-id>.<ext>` (e.g. a single-file `.html` or `.svg`).
2. **The registration** — `models/<model-id>.json` declaring it.
Do **not** touch anything else. In particular:
- **Do not edit `README.md` / `README.en.md`.** The Registry tables are
generated from `models/*.json` and synced automatically after merge. Hand
edits are overwritten. (**不要改 README**,注册表是自动生成并在合并后自动同步的。)
- **Do not edit `cases.json`.** Cases are maintainer-owned.(case 由维护者管理。)
## `models/<model-id>.json` schema
```jsonc
{
"label": "DeepSeek V4 Flash", // human-facing model name (required)
"vendor": "DeepSeek", // who makes the model (required)
"harness": "Reasonix", // the tool/agent/UI you ran it in (required)
"effort": "Max", // thinking budget / effort level (required)
"artifactDir": "deepseek-v4-flash/reasonix-max", // optional; defaults to <model-id>
"order": 50, // optional display sort
"runs": {
"skeleton-watch": {
"note": { // REQUIRED, bilingual, zh first, no emojis
"zh": "在 Reasonix 中以 DeepSeek V4 Flash(Max)一次生成,未修改。",
"en": "One-shot by DeepSeek V4 Flash (Max effort) in Reasonix, unmodified."
},
"contributor": "your-github-username"
}
}
}
```
### Naming rules / 命名规则
- **The filename IS the entry id** — lowercase letters, digits and dashes only
(`deepseek-v4-flash-reasonix`). It is the **stable voting key** and the default
`outputs/` folder, so name it after the **model**, not the harness. Bad:
`reasonix-reasonix-max`. Good: `deepseek-v4-flash-reasonix`.
(文件名即条目 id,是稳定投票主键——按**模型**命名,别按 harness 命名。)
- `artifactDir` may be one or two dash-case segments
(`deepseek-v4-flash/reasonix-max`); set it when several entries share a model
folder. Leave it out to default to the id.
### One entry per (model × harness × effort) / 一个组合一个条目
The same model under a different **harness** or **effort** is a **separate**
`models/*.json` file, not an extra variant inside an existing one. "Codex",
"Cursor", "Claude Code", "AntiGravity" name the **harness**, never the model.
(同一模型换 harness 或 effort = **新建** JSON,不要塞进现有条目。)
### The `note` is required and bilingual / note 必须双语
Every run needs `note.zh` + `note.en` documenting provenance: which harness,
which effort, one-shot or fixed. If you applied any fix, say where the bug was
and what you changed. **No emojis anywhere** (hard rule). zh first.
## Before you open the PR / 提交前
```bash
bun scripts/validate-data.ts # the exact schema check CI runs
```
This enforces: dash-only ids, required bilingual `note`, declared artifacts
exist and are non-empty, no orphan files under `outputs/`. Green here = green CI.
You do **not** need to run `update-registry.ts` — README is synced for you after
merge.(不需要手动跑 update-registry,README 合并后自动同步。)
### Attach a screenshot / 附一张截图
The PR template asks for a **screenshot of the generation** — your harness/IDE
showing the model name + effort while producing this artifact. It is the proof
that the declared `harness` / `model` / `effort` are real. Paste it into the PR
body (GitHub hosts it); **do not commit image files** into the repo.
(PR 里贴一张**生成过程截图**作为「确实用了这套 harness + 模型 + effort」的凭证,
直接贴进 PR 正文,不要把图片提交进仓库。)
## Conventions / 约定
- No emojis in any data, note, or UI-bound text.
- `outputs/` files are served verbatim in a sandboxed iframe. Keep them
self-contained; external resources should be well-known CDNs only.
+8 -6
View File
@@ -25,7 +25,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
## Registry
<!-- registry:start -->
> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI checks that it stays current.
> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI re-syncs it automatically after merge — contributors need not hand-edit it.
| Model | Vendor | Harness x Effort | Runs |
|---|---|---|---|
@@ -77,7 +77,7 @@ outputs/<artifact-dir>/<case-id>.<ext> raw model artifacts (HTML / SVG), filena
models/<agent-id>.json agent registry: label / vendor / harness / effort / artifactDir / order / run notes
cases.json case definitions (bilingual prompts, maintainer-owned)
scripts/validate-data.ts data validation (CI runs it on every PR)
scripts/update-registry.ts README Registry table generation (CI checks freshness)
scripts/update-registry.ts README Registry table generation (CI auto-syncs after merge)
```
> The site and vote-backend source lives in the private `nagi-bench-site` repo, deployed on Cloudflare Pages; this repo is the site's data source.
@@ -85,17 +85,19 @@ scripts/update-registry.ts README Registry table generation (CI checks
## Validation
```bash
bun scripts/update-registry.ts # updates README Registry tables from models/*.json
bun scripts/validate-data.ts # the same data validation CI runs
bun scripts/validate-data.ts # the same data validation CI runs (run this before submitting)
bun scripts/update-registry.ts # optional: preview the Registry tables locally; CI syncs them after merge
```
## Contributing a run
> Full rules live in [**AGENTS.md**](./AGENTS.md) (the contributor / AI-agent guide).
> **If you are an AI agent (Claude Code / Codex / Cursor, etc.), this part is for you:**
>
> - **The unit you contribute is one Agent = a model × the harness you run in.** The `<agent-id>` (lowercase letters / digits / dashes) encodes that pairing — e.g. `gpt-5-5-pro`, `claude-fable-5-cc` (cc = Claude Code). The same model in a different harness is a different Agent with a different id. The `<agent-id>` is the vote and leaderboard identity; do not rename it just to reorganize files.
> - **There are two cases right now**, defined in [`cases.json`](./cases.json): `mythos-craft` (HTML, a playable voxel world) and `pelican-cycling` (SVG, a pelican cycling by the sea). The prompt lives in `cases.json` — **use it verbatim, do not paraphrase**.
> - **To find what's missing**: read `cases.json` (all cases) and `models/*.json` (what each Agent has run) — a `<case-id>` absent from an Agent's json is an open slot; any row in the script-generated, CI-checked Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill both.
> - **There are three cases right now**, defined in [`cases.json`](./cases.json): `mythos-craft` (HTML, a playable voxel world), `pelican-cycling` (SVG, a pelican cycling by the sea), and `skeleton-watch` (HTML, a skeleton mechanical watch). The prompt lives in `cases.json` — **use it verbatim, do not paraphrase**.
> - **To find what's missing**: read `cases.json` (all cases) and `models/*.json` (what each Agent has run) — a `<case-id>` absent from an Agent's json is an open slot; any row in the script-generated Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill them.
> - **Then follow the two-file flow below**, write a bilingual `note` (how the artifact was produced — one-shot or fixed), and run `bun scripts/validate-data.ts` until it passes before opening a PR.
Contributions are data-only — no code changes needed:
+8 -6
View File
@@ -25,7 +25,7 @@ NAGI STUDIO 的 LLM 测评案例集:同一段提示词,不同「模型 × Ha
## 已测组合 / Registry
<!-- registry:start -->
> 此表由 `bun scripts/update-registry.ts` 从 `models/*.json` 自动生成;CI 会检查它是否最新。
> 此表由 `bun scripts/update-registry.ts` 从 `models/*.json` 生成;合并到 `main` 后由 CI 自动同步,贡献者无需手改。
| 模型 | 厂商 | 运行环境(Harness)× 思考配额 | 产出 |
|---|---|---|---|
@@ -77,7 +77,7 @@ outputs/<artifact-dir>/<case-id>.<ext> 模型产出原文件(HTML / SVG),
models/<agent-id>.json Agent 登记:label / vendor / harness / effort / artifactDir / order / 各案例运行备注
cases.json 案例定义:双语标题与提示词(维护者维护)
scripts/validate-data.ts 数据校验(CI 对每个 PR 自动执行)
scripts/update-registry.ts README Registry 表生成(CI 检查是否最新)
scripts/update-registry.ts README Registry 表生成(合并到 main 后由 CI 自动同步)
```
> 站点与投票后端的源码在私有仓库 `nagi-bench-site` 中,部署于 Cloudflare Pages;本仓库是站点的数据源。
@@ -85,17 +85,19 @@ scripts/update-registry.ts README Registry 表生成(CI 检查是否
## 本地校验 / Validation
```bash
bun scripts/update-registry.ts # 从 models/*.json 更新 README Registry 表
bun scripts/validate-data.ts # 与 CI 相同的数据校验
bun scripts/validate-data.ts # 与 CI 相同的数据校验(提交前跑这个即可)
bun scripts/update-registry.ts # 可选:本地预览 Registry 表;合并到 main 后 CI 会自动同步
```
## 贡献一个模型产出 / Contributing a run
> 完整规则见 [**AGENTS.md**](./AGENTS.md)(贡献者 / AI Agent 指南)。
> **如果你是一个 AI Agent(Claude Code / Codex / Cursor 等),这一段是给你看的:**
>
> - **你贡献的单位是一个 Agent = 模型 × 你所在的 Harness。** `<agent-id>`(小写字母 / 数字 / 连字符)就编码这个组合,如 `gpt-5-5-pro`、`claude-fable-5-cc`(cc = Claude Code)。同一模型换个 Harness 就是另一个 Agent、另一个 id。`<agent-id>` 是投票与榜单身份,不要为了整理文件夹而改它。
> - **当前有两个案例**,定义在 [`cases.json`](./cases.json):`mythos-craft`(HTML,可玩体素世界)与 `pelican-cycling`(SVG,海边骑车的鹈鹕)。提示词以 `cases.json` 为准,**逐字使用、不要改写**。
> - **想知道还缺哪些**:读 `cases.json`(全部案例)与 `models/*.json`(每个 Agent 跑过的案例)——某个 Agent 的 json 里缺某个 `<case-id>` 就是一个空位;上面由脚本生成并由 CI 检查的 Registry 表里,「产出」数小于案例总数的行,就是还缺案例的 Agent;全新 Agent 则两个案例都可补。
> - **当前有三个案例**,定义在 [`cases.json`](./cases.json):`mythos-craft`(HTML,可玩体素世界)、`pelican-cycling`(SVG,海边骑车的鹈鹕)与 `skeleton-watch`(HTML,镂空机械表)。提示词以 `cases.json` 为准,**逐字使用、不要改写**。
> - **想知道还缺哪些**:读 `cases.json`(全部案例)与 `models/*.json`(每个 Agent 跑过的案例)——某个 Agent 的 json 里缺某个 `<case-id>` 就是一个空位;上面由脚本生成的 Registry 表里,「产出」数小于案例总数的行,就是还缺案例的 Agent;全新 Agent 则三个案例都可补。
> - **然后照下面两文件流程做**,写好双语 `note`(说明产出怎么来的、是否一次生成、是否有修复),提交前用 `bun scripts/validate-data.ts` 自检通过再发 PR。
贡献**不需要改任何代码**,只涉及两类文件:
+2 -2
View File
@@ -85,8 +85,8 @@ function table(lang: 'zh' | 'en'): string {
const pending = groups.filter((group) => group.runs === 0).map((group) => escapeCell(group.label))
const note =
lang === 'zh'
? '> 此表由 `bun scripts/update-registry.ts` 从 `models/*.json` 自动生成;CI 会检查它是否最新。'
: '> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI checks that it stays current.'
? '> 此表由 `bun scripts/update-registry.ts` 从 `models/*.json` 生成;合并到 `main` 后由 CI 自动同步,贡献者无需手改。'
: '> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI re-syncs it automatically after merge — contributors need not hand-edit it.'
const header =
lang === 'zh'
? '| 模型 | 厂商 | 运行环境(Harness)× 思考配额 | 产出 |'