docs: require running the harness as it ships (no extra skills)

Address fairness concern from #24: the same harness can load completely
different skills / plugins / MCP / custom system prompts, which shifts the
output as much as swapping the harness and breaks Agent comparability.

- README.md / README.en.md: add a bullet to the Model + Harness section
  stating the harness is evaluated in its default, out-of-the-box form.
- AGENTS.md: add a "Run the harness as it ships" rule — no extra skills
  beyond what the harness bundles; disclose any deviation in `note`.

Enforcement stays honor-system + the PR generation screenshot.

Closes #24

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jesse ZhangandClaude Opus 4.8 committed 2026-06-24 20:18:08 +01:00
1 parent eaf2737ecc
commit 5cd30bac2b
3 files changed
+14

No files matched your search

+12
View File
@@ -71,6 +71,18 @@ Every run needs `note.zh` + `note.en` documenting provenance: which harness,
which effort, one-shot or fixed. If you applied any fix, say where the bug was
and what you changed. **No emojis anywhere** (hard rule). zh first.
### Run the harness as it ships / 用 harness 开箱即用的默认形态
Use the harness in its **default, out-of-the-box** configuration: no extra
skills, plugins, MCP servers or custom system prompts beyond what it bundles.
These shift the output as much as swapping the harness does, so loading them
makes the run no longer comparable to others under the same harness. If you used
any, it is a different setup — say so explicitly in `note`, otherwise it does
not belong on the board.
(用 harness 的**默认形态**:不要额外加载 harness 自带以外的 skill / 插件 /
MCP / 自定义系统提示——它们对产出的影响不亚于换 harness,会让这条 run 失去与
同一 harness 下其它条目的可比性。确实用了就在 `note` 里写清楚,否则不该上榜。)
## Before you open the PR / 提交前
```bash
+1
View File
@@ -21,6 +21,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination.
- **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
- The unit of evaluation here is therefore an **Agent**: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family).
- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. If you did use any, it is no longer the same agent; disclose it in the `note`.
## Registry
+1
View File
@@ -21,6 +21,7 @@ NAGI STUDIO 的 LLM 测评案例集:同一段提示词,不同「模型 × Ha
- **模型(Model)**:权重与推理引擎本身,如 GPT-5.5、Gemini 3.1 Pro、Claude Fable 5。思考配额(effort)是模型自身的能力旋钮——同一份权重在高/低思考强度下的推理深度判若两个模型,因此它和权重同属「模型」这一维度,脱离 effort 谈模型表现并不公允;本仓库默认拉满(Max),并在每个组合里如实标注。
- **Harness(运行环境)**:包裹模型的产品/脚手架,如 Codex CLI、Cursor、AntiGravity、Claude 网页版——它决定工具调用、系统提示词、上下文管理与续写策略,对最终产出的影响往往不亚于模型本身。
- 因此本仓库的测评单位是一个 **Agent**:同一个模型(含其思考配额)换一个 Harness,就是另一个 Agent,记为不同条目(例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5(xhigh)是同一模型家族的两个 Agent)。
- **测的是 Harness 开箱即用的默认形态**:请勿额外加载 Harness 自带以外的 skill / 插件 / MCP / 自定义系统提示——它们对产出的影响不亚于换 Harness,会破坏 Agent 之间的可比性。确实用了,就不是同一套 Agent,必须在 `note` 里如实说明。
## 已测组合 / Registry