feat: add cs-dust2 React case with two model takes + react kind support
New case cs-dust2 (kind:react): a from-scratch React + three.js 5v5 Dust2 FPS. Two takes registered: Claude Fable 5 / Claude Code / High (with a hand fix for a 64Hz fixed-step firing bug on high-refresh displays) and Claude Opus 4.8 / Claude Code / Max (dust2-arena, one-shot). React cases commit source only; the single file is built from source at deploy time. validate-data.ts now accepts a React project directory as the declared artifact. AGENTS.md rewritten English-only with React-case, bare-harness (no skills/plugins) and artifact-anonymity rules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jesse ZhangandClaude Opus 4.8 committed
1 parent
c4cc293748
commit
cbfce3b7cc
62 files changed
+13583
-43
No files matched your search
@@ -1,18 +1,15 @@
|
||||
# AGENTS.md
|
||||
|
||||
Guidance for humans and coding agents contributing to **NAGI BENCH** data
|
||||
(https://bench.nagi.fun/). Read this before opening a PR. 中文说明见每节。
|
||||
(https://bench.nagi.fun/). Read this before opening a PR.
|
||||
|
||||
## What this repo is / 这个仓库是什么
|
||||
## What this repo is
|
||||
|
||||
The **public data repo**: community-contributed *one-shot* LLM eval artifacts.
|
||||
You add a model's raw output for a case; the site renders it verbatim. This repo
|
||||
holds **data only** — no site code, no build of the page you see.
|
||||
|
||||
公开**数据仓库**：社区贡献的「一次生成」LLM 测评产物。你提交某模型在某个 case 上的
|
||||
原始输出，站点会原样渲染。这里只有**数据**，没有站点代码。
|
||||
|
||||
## A contribution is exactly two files / 一次贡献就两个文件
|
||||
## A contribution is exactly two files
|
||||
|
||||
1. **The artifact** — the raw model output, dropped at
|
||||
`outputs/<artifactDir>/<case-id>.<ext>` (e.g. a single-file `.html` or `.svg`).
|
||||
@@ -22,8 +19,8 @@ Do **not** touch anything else. In particular:
|
||||
|
||||
- **Do not edit `README.md` / `README.en.md`.** The Registry tables are
|
||||
generated from `models/*.json` and synced automatically after merge. Hand
|
||||
edits are overwritten. （**不要改 README**，注册表是自动生成并在合并后自动同步的。）
|
||||
- **Do not edit `cases.json`.** Cases are maintainer-owned.（case 由维护者管理。）
|
||||
edits are overwritten.
|
||||
- **Do not edit `cases.json`.** Cases are maintainer-owned.
|
||||
|
||||
## `models/<model-id>.json` schema
|
||||
|
||||
@@ -47,25 +44,23 @@ Do **not** touch anything else. In particular:
|
||||
}
|
||||
```
|
||||
|
||||
### Naming rules / 命名规则
|
||||
### Naming rules
|
||||
|
||||
- **The filename IS the entry id** — lowercase letters, digits and dashes only
|
||||
(`deepseek-v4-flash-reasonix`). It is the **stable voting key** and the default
|
||||
`outputs/` folder, so name it after the **model**, not the harness. Bad:
|
||||
`reasonix-reasonix-max`. Good: `deepseek-v4-flash-reasonix`.
|
||||
（文件名即条目 id，是稳定投票主键——按**模型**命名，别按 harness 命名。）
|
||||
- `artifactDir` may be one or two dash-case segments
|
||||
(`deepseek-v4-flash/reasonix-max`); set it when several entries share a model
|
||||
folder. Leave it out to default to the id.
|
||||
|
||||
### One entry per (model × harness × effort) / 一个组合一个条目
|
||||
### One entry per (model × harness × effort)
|
||||
|
||||
The same model under a different **harness** or **effort** is a **separate**
|
||||
`models/*.json` file, not an extra variant inside an existing one. "Codex",
|
||||
"Cursor", "Claude Code", "AntiGravity" name the **harness**, never the model.
|
||||
（同一模型换 harness 或 effort = **新建** JSON，不要塞进现有条目。）
|
||||
|
||||
### Multiple takes of one case / 同一 case 留多版
|
||||
### Multiple takes of one case
|
||||
|
||||
Ran the **same** model × harness × effort on a case more than once and want to
|
||||
keep several outputs? Make `runs.<case-id>` an **array** — this is the *only*
|
||||
@@ -89,30 +84,51 @@ still a new file, per above):
|
||||
- Each variant carries its own `note` and may set its own `contributor`, so
|
||||
different people can add takes to the same model entry.
|
||||
|
||||
（同一 **model × harness × effort** 想留多张，就把 `runs.<case>` 写成**数组**，别新建
|
||||
JSON——这是唯一能把多份产物塞进一个条目的情形。第一版可省略 `file`，其余每版都要有
|
||||
各自不同、描述性的 dash-case `file` 名；**别用 `index.html`**（会被当成目录默认页）。
|
||||
每版各自带 note，可各自署名 contributor。）
|
||||
|
||||
### The `note` is required and bilingual / note 必须双语
|
||||
### The `note` is required and bilingual
|
||||
|
||||
Every run needs `note.zh` + `note.en` documenting provenance: which harness,
|
||||
which effort, one-shot or fixed. If you applied any fix, say where the bug was
|
||||
and what you changed. **No emojis anywhere** (hard rule). zh first.
|
||||
|
||||
### Run the harness as it ships / 用 harness 开箱即用的默认形态
|
||||
### Run the bare harness — NO skills, NO plugins
|
||||
|
||||
Use the harness in its **default, out-of-the-box** configuration: no extra
|
||||
skills, plugins, MCP servers or custom system prompts beyond what it bundles.
|
||||
These shift the output as much as swapping the harness does, so such a run is
|
||||
not comparable to others under the same harness and **will not be accepted**.
|
||||
Disclosing it in `note` does not fix this: the leaderboard identity is only
|
||||
model × harness × effort, so a note gives the modified setup no separate
|
||||
identity — it would still be mixed into an Agent row it is not comparable with.
|
||||
（用 harness 的**默认形态**：不要额外加载 harness 自带以外的 skill / 插件 /
|
||||
MCP / 自定义系统提示——它们对产出的影响不亚于换 harness，这类 run 与同一
|
||||
harness 下其它条目不可比，**不予收录**。在 `note` 里写明也没用：榜单身份只看
|
||||
模型 × harness × effort，note 不另立身份，它仍会混进本不该可比的那一行。）
|
||||
Run the model with the harness's default model and effort, but otherwise
|
||||
**stripped down**: **disable every skill and every plugin — including the ones
|
||||
the harness ships with and would auto-invoke** — and add no MCP servers and no
|
||||
custom system prompt. A skill or plugin (a built-in `design`, `gsap`, game, or
|
||||
diagram helper, an added toolset, etc.) rewrites the output far more than the
|
||||
model itself does, so a run with any of them active measures the tooling, not the
|
||||
model × harness × effort. It is not comparable to other entries and **will not be
|
||||
accepted**. Disclosing it in `note` does not fix this: leaderboard identity is
|
||||
only model × harness × effort, so a modified setup gets no separate identity — it
|
||||
would just be mixed into a row it is not comparable with.
|
||||
|
||||
### Keep the artifact anonymous — no identity leaks
|
||||
|
||||
**The output must not reveal which model, vendor, or harness produced it.** Strip
|
||||
any `<!-- Generated by ... -->` header, watermark, "as an AI / as a large
|
||||
language model" phrasing, or signature in visible text or code comments before
|
||||
you submit. Blind Arena battles depend on outputs being anonymous, so a
|
||||
self-identifying artifact invalidates the entry.
|
||||
|
||||
## React cases — contribute `src/` only
|
||||
|
||||
Some cases (marked `kind: "react"` in `cases.json`) ask for a real multi-file
|
||||
React project instead of a single file. For these your artifact is **the `src/`
|
||||
tree only** — multiple files and nested folders are fine — dropped at
|
||||
`outputs/<artifactDir>/src/`. The site owns the whole toolchain and builds every
|
||||
React case with **one shared scaffold** (its own installed React + Vite), so
|
||||
there is no per-project dependency setup:
|
||||
|
||||
- **Do NOT include `package.json`, a lockfile, `node_modules`, `index.html`, or
|
||||
any build / Vite / TS config.** The scaffold supplies all of it.
|
||||
- **Use only libraries the site already ships** (React + ReactDOM, plus whatever
|
||||
is in its `package.json`). The build installs nothing new — you cannot pull in
|
||||
arbitrary npm packages.
|
||||
- **The entry point is `src/main.tsx`**, mounting your app into `<div id="root">`.
|
||||
- At build time the `src/` is bundled and inlined into **one self-contained HTML
|
||||
file** that the normal viewer serves verbatim. The anonymity rule above still
|
||||
applies: no model / vendor / harness identity anywhere in the code.
|
||||
|
||||
## Review guidelines
|
||||
|
||||
@@ -133,7 +149,7 @@ Do not ask contributors to improve or repair the submitted artifacts themselves.
|
||||
If an artifact bug matters, mention it only as optional context for maintainers,
|
||||
not as required PR feedback.
|
||||
|
||||
## Before you open the PR / 提交前
|
||||
## Before you open the PR
|
||||
|
||||
```bash
|
||||
bun scripts/validate-data.ts # the exact schema check CI runs
|
||||
@@ -143,18 +159,16 @@ This enforces: dash-only ids, required bilingual `note`, declared artifacts
|
||||
exist and are non-empty, no orphan files under `outputs/`. Green here = green CI.
|
||||
|
||||
You do **not** need to run `update-registry.ts` — README is synced for you after
|
||||
merge.（不需要手动跑 update-registry，README 合并后自动同步。）
|
||||
merge.
|
||||
|
||||
### Attach a screenshot / 附一张截图
|
||||
### Attach a screenshot
|
||||
|
||||
The PR template asks for a **screenshot of the generation** — your harness/IDE
|
||||
showing the model name + effort while producing this artifact. It is the proof
|
||||
that the declared `harness` / `model` / `effort` are real. Paste it into the PR
|
||||
body (GitHub hosts it); **do not commit image files** into the repo.
|
||||
（PR 里贴一张**生成过程截图**作为「确实用了这套 harness + 模型 + effort」的凭证，
|
||||
直接贴进 PR 正文，不要把图片提交进仓库。）
|
||||
|
||||
## Conventions / 约定
|
||||
## Conventions
|
||||
|
||||
- No emojis in any data, note, or UI-bound text.
|
||||
- `outputs/` files are served verbatim in a sandboxed iframe. Keep them
|
||||
|
||||
Reference in new issue
Block a user