Add AGENTS.md + tighten contribution flow
- AGENTS.md: public contributor/agent guide (two-file rule, model-centric ids, one-entry-per-harness, bilingual notes, screenshot proof) - PR template: require a generation proof screenshot; drop the manual update-registry step; add id-naming and no-README-edit checks - CI: PRs only run validate-data; pushes to main auto-regenerate and commit the README registry (contents:write on that job, [skip ci], no loop) - README/update-registry: reflect auto-synced registry; fix stale 'two cases' -> three (mythos-craft, pelican-cycling, skeleton-watch); link AGENTS.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
2fb8499fa8
commit
30983f0e7d
6 files changed
+167
-26
No files matched your search
@@ -0,0 +1,99 @@
|
||||
# AGENTS.md
|
||||
|
||||
Guidance for humans and coding agents contributing to **NAGI BENCH** data
|
||||
(https://bench.nagi.fun/). Read this before opening a PR. 中文说明见每节。
|
||||
|
||||
## What this repo is / 这个仓库是什么
|
||||
|
||||
The **public data repo**: community-contributed *one-shot* LLM eval artifacts.
|
||||
You add a model's raw output for a case; the site renders it verbatim. This repo
|
||||
holds **data only** — no site code, no build of the page you see.
|
||||
|
||||
公开**数据仓库**:社区贡献的「一次生成」LLM 测评产物。你提交某模型在某个 case 上的
|
||||
原始输出,站点会原样渲染。这里只有**数据**,没有站点代码。
|
||||
|
||||
## A contribution is exactly two files / 一次贡献就两个文件
|
||||
|
||||
1. **The artifact** — the raw model output, dropped at
|
||||
`outputs/<artifactDir>/<case-id>.<ext>` (e.g. a single-file `.html` or `.svg`).
|
||||
2. **The registration** — `models/<model-id>.json` declaring it.
|
||||
|
||||
Do **not** touch anything else. In particular:
|
||||
|
||||
- **Do not edit `README.md` / `README.en.md`.** The Registry tables are
|
||||
generated from `models/*.json` and synced automatically after merge. Hand
|
||||
edits are overwritten. (**不要改 README**,注册表是自动生成并在合并后自动同步的。)
|
||||
- **Do not edit `cases.json`.** Cases are maintainer-owned.(case 由维护者管理。)
|
||||
|
||||
## `models/<model-id>.json` schema
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"label": "DeepSeek V4 Flash", // human-facing model name (required)
|
||||
"vendor": "DeepSeek", // who makes the model (required)
|
||||
"harness": "Reasonix", // the tool/agent/UI you ran it in (required)
|
||||
"effort": "Max", // thinking budget / effort level (required)
|
||||
"artifactDir": "deepseek-v4-flash/reasonix-max", // optional; defaults to <model-id>
|
||||
"order": 50, // optional display sort
|
||||
"runs": {
|
||||
"skeleton-watch": {
|
||||
"note": { // REQUIRED, bilingual, zh first, no emojis
|
||||
"zh": "在 Reasonix 中以 DeepSeek V4 Flash(Max)一次生成,未修改。",
|
||||
"en": "One-shot by DeepSeek V4 Flash (Max effort) in Reasonix, unmodified."
|
||||
},
|
||||
"contributor": "your-github-username"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Naming rules / 命名规则
|
||||
|
||||
- **The filename IS the entry id** — lowercase letters, digits and dashes only
|
||||
(`deepseek-v4-flash-reasonix`). It is the **stable voting key** and the default
|
||||
`outputs/` folder, so name it after the **model**, not the harness. Bad:
|
||||
`reasonix-reasonix-max`. Good: `deepseek-v4-flash-reasonix`.
|
||||
(文件名即条目 id,是稳定投票主键——按**模型**命名,别按 harness 命名。)
|
||||
- `artifactDir` may be one or two dash-case segments
|
||||
(`deepseek-v4-flash/reasonix-max`); set it when several entries share a model
|
||||
folder. Leave it out to default to the id.
|
||||
|
||||
### One entry per (model × harness × effort) / 一个组合一个条目
|
||||
|
||||
The same model under a different **harness** or **effort** is a **separate**
|
||||
`models/*.json` file, not an extra variant inside an existing one. "Codex",
|
||||
"Cursor", "Claude Code", "AntiGravity" name the **harness**, never the model.
|
||||
(同一模型换 harness 或 effort = **新建** JSON,不要塞进现有条目。)
|
||||
|
||||
### The `note` is required and bilingual / note 必须双语
|
||||
|
||||
Every run needs `note.zh` + `note.en` documenting provenance: which harness,
|
||||
which effort, one-shot or fixed. If you applied any fix, say where the bug was
|
||||
and what you changed. **No emojis anywhere** (hard rule). zh first.
|
||||
|
||||
## Before you open the PR / 提交前
|
||||
|
||||
```bash
|
||||
bun scripts/validate-data.ts # the exact schema check CI runs
|
||||
```
|
||||
|
||||
This enforces: dash-only ids, required bilingual `note`, declared artifacts
|
||||
exist and are non-empty, no orphan files under `outputs/`. Green here = green CI.
|
||||
|
||||
You do **not** need to run `update-registry.ts` — README is synced for you after
|
||||
merge.(不需要手动跑 update-registry,README 合并后自动同步。)
|
||||
|
||||
### Attach a screenshot / 附一张截图
|
||||
|
||||
The PR template asks for a **screenshot of the generation** — your harness/IDE
|
||||
showing the model name + effort while producing this artifact. It is the proof
|
||||
that the declared `harness` / `model` / `effort` are real. Paste it into the PR
|
||||
body (GitHub hosts it); **do not commit image files** into the repo.
|
||||
(PR 里贴一张**生成过程截图**作为「确实用了这套 harness + 模型 + effort」的凭证,
|
||||
直接贴进 PR 正文,不要把图片提交进仓库。)
|
||||
|
||||
## Conventions / 约定
|
||||
|
||||
- No emojis in any data, note, or UI-bound text.
|
||||
- `outputs/` files are served verbatim in a sandboxed iframe. Keep them
|
||||
self-contained; external resources should be well-known CDNs only.
|
||||
Reference in new issue
Block a user