diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 7feffe4..744532c 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,19 +1,33 @@ - - + + ## 产出说明 / Run info - 模型 / Model: -- 运行环境 / Harness（如 ChatGPT Web、Claude Code、Google AI Studio、本地 CLI…）: -- 思考配额 / Thinking effort（如 Max、High、Extended Pro、Thinking…）: -- 是否一次生成 / One-shot? （是/否；若有修复请在 note 里说明 / if fixed, document it in the note） +- 厂商 / Vendor: +- 运行环境 / Harness（如 ChatGPT Web、Claude Code、Codex CLI、AntiGravity…，这是工具不是模型）: +- 思考配额 / Thinking effort（如 Max、High、xhigh、Extended Pro、Thinking…）: +- 是否一次生成 / One-shot?（是/否；若有修复请在 note 里说明 / if fixed, document it in the note） + +## 生成凭证截图 / Proof screenshot **(required)** + + + + +请贴一张**生成过程截图**：你的 harness / IDE 中能看到**模型名 + 思考配额**正在产出本产物， +作为「确实用了这套 harness + 模型 + effort」的凭证。 +Please attach a screenshot of the generation showing the **model name + effort** in your harness/IDE. ## 自查 / Checklist -- [ ] `models/.json` 中 `harness` 与 `effort` 已如实填写（站点据此生成 metadata 与品牌 icon） -- [ ] 每条 run 的 `note` 双语（zh + en）填写了上面的来历信息 / bilingual `note` documents the provenance above +- [ ] 只改了两类文件：`outputs//.` + `models/.json` / only the artifact + its JSON +- [ ] **没有**手改 `README.md` / `README.en.md`（注册表合并后由 CI 自动同步）/ did **not** hand-edit the README registry (auto-synced after merge) +- [ ] 模型 id 用短横线、以**模型**命名（非 harness），如 `deepseek-v4-flash-reasonix` / dash-case id named after the model, not the harness +- [ ] `harness` 与 `effort` 如实填写（站点据此生成 metadata 与品牌 icon）/ `harness` and `effort` are accurate +- [ ] 每条 run 的 `note` 双语（zh + en，zh 先，无 emoji）填了来历 / bilingual `note`, zh first, no emojis +- [ ] 同一模型换 Harness 或思考配额是**新建一个 JSON 条目** / a different harness or effort = a NEW json entry - [ ] `contributor` 是我的 GitHub 用户名 / `contributor` is my GitHub username -- [ ] 同一模型在不同 Harness 或思考配额下是**新建一个 JSON 条目**，不是塞进现有条目 / a different harness or effort = a NEW json entry, not a variant -- [ ] 已运行 `bun scripts/update-registry.ts` 更新 README Registry 表 / ran `bun scripts/update-registry.ts` to refresh the README Registry tables +- [ ] 已贴生成凭证截图 / attached the proof screenshot above +- [ ] 本地 `bun scripts/validate-data.ts` 通过 / `bun scripts/validate-data.ts` passes locally - + diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bca6c79..33096ef 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -14,11 +14,35 @@ jobs: steps: - uses: actions/checkout@v4 - uses: oven-sh/setup-bun@v2 - - name: Check README registry - run: bun scripts/update-registry.ts --check - name: Validate contribution data run: bun scripts/validate-data.ts + sync-registry: + # The README Registry tables are generated from models/*.json. Contributors + # do not edit them by hand; on every push to main we regenerate and commit + # any drift back. Commits pushed with GITHUB_TOKEN do not retrigger + # workflows, and the message carries [skip ci] too, so there is no loop. + if: github.event_name == 'push' + needs: validate + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - uses: actions/checkout@v4 + - uses: oven-sh/setup-bun@v2 + - name: Regenerate README registry + run: bun scripts/update-registry.ts + - name: Commit if registry drifted + run: | + if git diff --quiet -- README.md README.en.md; then + echo "Registry already in sync." + exit 0 + fi + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git commit -am "chore: sync README registry [skip ci]" + git push origin HEAD:main + trigger-site-rebuild: # The site lives in the private nagi-bench-site repo (Cloudflare Pages). # With SITE_DISPATCH_TOKEN set the rebuild is instant; without it the diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..4e4d617 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,99 @@ +# AGENTS.md + +Guidance for humans and coding agents contributing to **NAGI BENCH** data +(https://bench.nagi.fun/). Read this before opening a PR. 中文说明见每节。 + +## What this repo is / 这个仓库是什么 + +The **public data repo**: community-contributed *one-shot* LLM eval artifacts. +You add a model's raw output for a case; the site renders it verbatim. This repo +holds **data only** — no site code, no build of the page you see. + +公开**数据仓库**：社区贡献的「一次生成」LLM 测评产物。你提交某模型在某个 case 上的 +原始输出，站点会原样渲染。这里只有**数据**，没有站点代码。 + +## A contribution is exactly two files / 一次贡献就两个文件 + +1. **The artifact** — the raw model output, dropped at + `outputs//.` (e.g. a single-file `.html` or `.svg`). +2. **The registration** — `models/.json` declaring it. + +Do **not** touch anything else. In particular: + +- **Do not edit `README.md` / `README.en.md`.** The Registry tables are + generated from `models/*.json` and synced automatically after merge. Hand + edits are overwritten. （**不要改 README**，注册表是自动生成并在合并后自动同步的。） +- **Do not edit `cases.json`.** Cases are maintainer-owned.（case 由维护者管理。） + +## `models/.json` schema + +```jsonc +{ + "label": "DeepSeek V4 Flash", // human-facing model name (required) + "vendor": "DeepSeek", // who makes the model (required) + "harness": "Reasonix", // the tool/agent/UI you ran it in (required) + "effort": "Max", // thinking budget / effort level (required) + "artifactDir": "deepseek-v4-flash/reasonix-max", // optional; defaults to + "order": 50, // optional display sort + "runs": { + "skeleton-watch": { + "note": { // REQUIRED, bilingual, zh first, no emojis + "zh": "在 Reasonix 中以 DeepSeek V4 Flash（Max）一次生成，未修改。", + "en": "One-shot by DeepSeek V4 Flash (Max effort) in Reasonix, unmodified." + }, + "contributor": "your-github-username" + } + } +} +``` + +### Naming rules / 命名规则 + +- **The filename IS the entry id** — lowercase letters, digits and dashes only + (`deepseek-v4-flash-reasonix`). It is the **stable voting key** and the default + `outputs/` folder, so name it after the **model**, not the harness. Bad: + `reasonix-reasonix-max`. Good: `deepseek-v4-flash-reasonix`. + （文件名即条目 id，是稳定投票主键——按**模型**命名，别按 harness 命名。） +- `artifactDir` may be one or two dash-case segments + (`deepseek-v4-flash/reasonix-max`); set it when several entries share a model + folder. Leave it out to default to the id. + +### One entry per (model × harness × effort) / 一个组合一个条目 + +The same model under a different **harness** or **effort** is a **separate** +`models/*.json` file, not an extra variant inside an existing one. "Codex", +"Cursor", "Claude Code", "AntiGravity" name the **harness**, never the model. +（同一模型换 harness 或 effort = **新建** JSON，不要塞进现有条目。） + +### The `note` is required and bilingual / note 必须双语 + +Every run needs `note.zh` + `note.en` documenting provenance: which harness, +which effort, one-shot or fixed. If you applied any fix, say where the bug was +and what you changed. **No emojis anywhere** (hard rule). zh first. + +## Before you open the PR / 提交前 + +```bash +bun scripts/validate-data.ts # the exact schema check CI runs +``` + +This enforces: dash-only ids, required bilingual `note`, declared artifacts +exist and are non-empty, no orphan files under `outputs/`. Green here = green CI. + +You do **not** need to run `update-registry.ts` — README is synced for you after +merge.（不需要手动跑 update-registry，README 合并后自动同步。） + +### Attach a screenshot / 附一张截图 + +The PR template asks for a **screenshot of the generation** — your harness/IDE +showing the model name + effort while producing this artifact. It is the proof +that the declared `harness` / `model` / `effort` are real. Paste it into the PR +body (GitHub hosts it); **do not commit image files** into the repo. +（PR 里贴一张**生成过程截图**作为「确实用了这套 harness + 模型 + effort」的凭证， +直接贴进 PR 正文，不要把图片提交进仓库。） + +## Conventions / 约定 + +- No emojis in any data, note, or UI-bound text. +- `outputs/` files are served verbatim in a sandboxed iframe. Keep them + self-contained; external resources should be well-known CDNs only. diff --git a/README.en.md b/README.en.md index 0413903..a0e3c78 100644 --- a/README.en.md +++ b/README.en.md @@ -25,7 +25,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x ## Registry -> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI checks that it stays current. +> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI re-syncs it automatically after merge — contributors need not hand-edit it. | Model | Vendor | Harness x Effort | Runs | |---|---|---|---| @@ -77,7 +77,7 @@ outputs//. raw model artifacts (HTML / SVG), filena models/.json agent registry: label / vendor / harness / effort / artifactDir / order / run notes cases.json case definitions (bilingual prompts, maintainer-owned) scripts/validate-data.ts data validation (CI runs it on every PR) -scripts/update-registry.ts README Registry table generation (CI checks freshness) +scripts/update-registry.ts README Registry table generation (CI auto-syncs after merge) ``` > The site and vote-backend source lives in the private `nagi-bench-site` repo, deployed on Cloudflare Pages; this repo is the site's data source. @@ -85,17 +85,19 @@ scripts/update-registry.ts README Registry table generation (CI checks ## Validation ```bash -bun scripts/update-registry.ts # updates README Registry tables from models/*.json -bun scripts/validate-data.ts # the same data validation CI runs +bun scripts/validate-data.ts # the same data validation CI runs (run this before submitting) +bun scripts/update-registry.ts # optional: preview the Registry tables locally; CI syncs them after merge ``` ## Contributing a run +> Full rules live in [**AGENTS.md**](./AGENTS.md) (the contributor / AI-agent guide). + > **If you are an AI agent (Claude Code / Codex / Cursor, etc.), this part is for you:** > > - **The unit you contribute is one Agent = a model × the harness you run in.** The `` (lowercase letters / digits / dashes) encodes that pairing — e.g. `gpt-5-5-pro`, `claude-fable-5-cc` (cc = Claude Code). The same model in a different harness is a different Agent with a different id. The `` is the vote and leaderboard identity; do not rename it just to reorganize files. -> - **There are two cases right now**, defined in [`cases.json`](./cases.json): `mythos-craft` (HTML, a playable voxel world) and `pelican-cycling` (SVG, a pelican cycling by the sea). The prompt lives in `cases.json` — **use it verbatim, do not paraphrase**. -> - **To find what's missing**: read `cases.json` (all cases) and `models/*.json` (what each Agent has run) — a `` absent from an Agent's json is an open slot; any row in the script-generated, CI-checked Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill both. +> - **There are three cases right now**, defined in [`cases.json`](./cases.json): `mythos-craft` (HTML, a playable voxel world), `pelican-cycling` (SVG, a pelican cycling by the sea), and `skeleton-watch` (HTML, a skeleton mechanical watch). The prompt lives in `cases.json` — **use it verbatim, do not paraphrase**. +> - **To find what's missing**: read `cases.json` (all cases) and `models/*.json` (what each Agent has run) — a `` absent from an Agent's json is an open slot; any row in the script-generated Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill them. > - **Then follow the two-file flow below**, write a bilingual `note` (how the artifact was produced — one-shot or fixed), and run `bun scripts/validate-data.ts` until it passes before opening a PR. Contributions are data-only — no code changes needed: diff --git a/README.md b/README.md index 7e0644c..9cfcab9 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ NAGI STUDIO 的 LLM 测评案例集：同一段提示词，不同「模型 × Ha ## 已测组合 / Registry -> 此表由 `bun scripts/update-registry.ts` 从 `models/*.json` 自动生成；CI 会检查它是否最新。 +> 此表由 `bun scripts/update-registry.ts` 从 `models/*.json` 生成；合并到 `main` 后由 CI 自动同步，贡献者无需手改。 | 模型 | 厂商 | 运行环境（Harness）× 思考配额 | 产出 | |---|---|---|---| @@ -77,7 +77,7 @@ outputs//. 模型产出原文件（HTML / SVG）， models/.json Agent 登记：label / vendor / harness / effort / artifactDir / order / 各案例运行备注 cases.json 案例定义：双语标题与提示词（维护者维护） scripts/validate-data.ts 数据校验（CI 对每个 PR 自动执行） -scripts/update-registry.ts README Registry 表生成（CI 检查是否最新） +scripts/update-registry.ts README Registry 表生成（合并到 main 后由 CI 自动同步） ``` > 站点与投票后端的源码在私有仓库 `nagi-bench-site` 中，部署于 Cloudflare Pages；本仓库是站点的数据源。 @@ -85,17 +85,19 @@ scripts/update-registry.ts README Registry 表生成（CI 检查是否 ## 本地校验 / Validation ```bash -bun scripts/update-registry.ts # 从 models/*.json 更新 README Registry 表 -bun scripts/validate-data.ts # 与 CI 相同的数据校验 +bun scripts/validate-data.ts # 与 CI 相同的数据校验（提交前跑这个即可） +bun scripts/update-registry.ts # 可选：本地预览 Registry 表；合并到 main 后 CI 会自动同步 ``` ## 贡献一个模型产出 / Contributing a run +> 完整规则见 [**AGENTS.md**](./AGENTS.md)（贡献者 / AI Agent 指南）。 + > **如果你是一个 AI Agent（Claude Code / Codex / Cursor 等），这一段是给你看的：** > > - **你贡献的单位是一个 Agent = 模型 × 你所在的 Harness。** ``（小写字母 / 数字 / 连字符）就编码这个组合，如 `gpt-5-5-pro`、`claude-fable-5-cc`（cc = Claude Code）。同一模型换个 Harness 就是另一个 Agent、另一个 id。`` 是投票与榜单身份，不要为了整理文件夹而改它。 -> - **当前有两个案例**，定义在 [`cases.json`](./cases.json)：`mythos-craft`（HTML，可玩体素世界）与 `pelican-cycling`（SVG，海边骑车的鹈鹕）。提示词以 `cases.json` 为准，**逐字使用、不要改写**。 -> - **想知道还缺哪些**：读 `cases.json`（全部案例）与 `models/*.json`（每个 Agent 跑过的案例）——某个 Agent 的 json 里缺某个 `` 就是一个空位；上面由脚本生成并由 CI 检查的 Registry 表里，「产出」数小于案例总数的行，就是还缺案例的 Agent；全新 Agent 则两个案例都可补。 +> - **当前有三个案例**，定义在 [`cases.json`](./cases.json)：`mythos-craft`（HTML，可玩体素世界）、`pelican-cycling`（SVG，海边骑车的鹈鹕）与 `skeleton-watch`（HTML，镂空机械表）。提示词以 `cases.json` 为准，**逐字使用、不要改写**。 +> - **想知道还缺哪些**：读 `cases.json`（全部案例）与 `models/*.json`（每个 Agent 跑过的案例）——某个 Agent 的 json 里缺某个 `` 就是一个空位；上面由脚本生成的 Registry 表里，「产出」数小于案例总数的行，就是还缺案例的 Agent；全新 Agent 则三个案例都可补。 > - **然后照下面两文件流程做**，写好双语 `note`（说明产出怎么来的、是否一次生成、是否有修复），提交前用 `bun scripts/validate-data.ts` 自检通过再发 PR。 贡献**不需要改任何代码**，只涉及两类文件： diff --git a/scripts/update-registry.ts b/scripts/update-registry.ts index 8a59606..fe5a57b 100644 --- a/scripts/update-registry.ts +++ b/scripts/update-registry.ts @@ -85,8 +85,8 @@ function table(lang: 'zh' | 'en'): string { const pending = groups.filter((group) => group.runs === 0).map((group) => escapeCell(group.label)) const note = lang === 'zh' - ? '> 此表由 `bun scripts/update-registry.ts` 从 `models/*.json` 自动生成；CI 会检查它是否最新。' - : '> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI checks that it stays current.' + ? '> 此表由 `bun scripts/update-registry.ts` 从 `models/*.json` 生成；合并到 `main` 后由 CI 自动同步，贡献者无需手改。' + : '> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI re-syncs it automatically after merge — contributors need not hand-edit it.' const header = lang === 'zh' ? '| 模型 | 厂商 | 运行环境（Harness）× 思考配额 | 产出 |'