docs: reject runs with extra skills outright (addr Codex #25 P2)

A `note` disclosure does not separate a modified-harness run on the board:
leaderboard identity is only model × harness × effort, so a skill-boosted run
would still land in an Agent row it is explicitly not comparable with. Drop the
"disclose in note" escape hatch and make it a flat non-acceptance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jesse ZhangandClaude Opus 4.8 committed 2026-06-24 20:25:08 +01:00
1 parent 5cd30bac2b
commit 308c93ac81
3 files changed
+10 -8

No files matched your search

+8 -6
View File
@@ -75,13 +75,15 @@ and what you changed. **No emojis anywhere** (hard rule). zh first.
Use the harness in its **default, out-of-the-box** configuration: no extra
skills, plugins, MCP servers or custom system prompts beyond what it bundles.
These shift the output as much as swapping the harness does, so loading them
makes the run no longer comparable to others under the same harness. If you used
any, it is a different setup — say so explicitly in `note`, otherwise it does
not belong on the board.
These shift the output as much as swapping the harness does, so such a run is
not comparable to others under the same harness and **will not be accepted**.
Disclosing it in `note` does not fix this: the leaderboard identity is only
model × harness × effort, so a note gives the modified setup no separate
identity — it would still be mixed into an Agent row it is not comparable with.
(用 harness 的**默认形态**:不要额外加载 harness 自带以外的 skill / 插件 /
MCP / 自定义系统提示——它们对产出的影响不亚于换 harness,会让这条 run 失去与
同一 harness 下其它条目的可比性。确实用了就在 `note` 里写清楚,否则不该上榜。)
MCP / 自定义系统提示——它们对产出的影响不亚于换 harness,这类 run 与同一
harness 下其它条目不可比,**不予收录**。在 `note` 里写明也没用:榜单身份只看
模型 × harness × effort,note 不另立身份,它仍会混进本不该可比的那一行。)
## Before you open the PR / 提交前
+1 -1
View File
@@ -21,7 +21,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination.
- **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
- The unit of evaluation here is therefore an **Agent**: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family).
- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. If you did use any, it is no longer the same agent; disclose it in the `note`.
- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. Such runs are **not accepted** (the leaderboard identity is just model × harness × effort, so a `note` does not give them a separate identity).
## Registry
+1 -1
View File
@@ -21,7 +21,7 @@ NAGI STUDIO 的 LLM 测评案例集:同一段提示词,不同「模型 × Ha
- **模型(Model)**:权重与推理引擎本身,如 GPT-5.5、Gemini 3.1 Pro、Claude Fable 5。思考配额(effort)是模型自身的能力旋钮——同一份权重在高/低思考强度下的推理深度判若两个模型,因此它和权重同属「模型」这一维度,脱离 effort 谈模型表现并不公允;本仓库默认拉满(Max),并在每个组合里如实标注。
- **Harness(运行环境)**:包裹模型的产品/脚手架,如 Codex CLI、Cursor、AntiGravity、Claude 网页版——它决定工具调用、系统提示词、上下文管理与续写策略,对最终产出的影响往往不亚于模型本身。
- 因此本仓库的测评单位是一个 **Agent**:同一个模型(含其思考配额)换一个 Harness,就是另一个 Agent,记为不同条目(例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5(xhigh)是同一模型家族的两个 Agent)。
- **测的是 Harness 开箱即用的默认形态**:请勿额外加载 Harness 自带以外的 skill / 插件 / MCP / 自定义系统提示——它们对产出的影响不亚于换 Harness,会破坏 Agent 之间的可比性。确实用了,就不是同一套 Agent,必须在 `note` 里如实说明。
- **测的是 Harness 开箱即用的默认形态**:请勿额外加载 Harness 自带以外的 skill / 插件 / MCP / 自定义系统提示——它们对产出的影响不亚于换 Harness,会破坏 Agent 之间的可比性。这类 run **不予收录**(榜单身份只看 模型 × Harness × effort,在 `note` 里说明也不另立身份)。
## 已测组合 / Registry