From 308c93ac81ebcf1d4c2c4e95bb71a7dee51a90f7 Mon Sep 17 00:00:00 2001 From: Jesse Zhang <101612750+Nagi-ovo@users.noreply.github.com> Date: Wed, 24 Jun 2026 20:25:08 +0100 Subject: [PATCH] docs: reject runs with extra skills outright (addr Codex #25 P2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `note` disclosure does not separate a modified-harness run on the board: leaderboard identity is only model × harness × effort, so a skill-boosted run would still land in an Agent row it is explicitly not comparable with. Drop the "disclose in note" escape hatch and make it a flat non-acceptance. Co-Authored-By: Claude Opus 4.8 (1M context) --- AGENTS.md | 14 ++++++++------ README.en.md | 2 +- README.md | 2 +- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a749f8b..f6b719d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,13 +75,15 @@ and what you changed. **No emojis anywhere** (hard rule). zh first. Use the harness in its **default, out-of-the-box** configuration: no extra skills, plugins, MCP servers or custom system prompts beyond what it bundles. -These shift the output as much as swapping the harness does, so loading them -makes the run no longer comparable to others under the same harness. If you used -any, it is a different setup — say so explicitly in `note`, otherwise it does -not belong on the board. +These shift the output as much as swapping the harness does, so such a run is +not comparable to others under the same harness and **will not be accepted**. +Disclosing it in `note` does not fix this: the leaderboard identity is only +model × harness × effort, so a note gives the modified setup no separate +identity — it would still be mixed into an Agent row it is not comparable with. （用 harness 的**默认形态**：不要额外加载 harness 自带以外的 skill / 插件 / -MCP / 自定义系统提示——它们对产出的影响不亚于换 harness，会让这条 run 失去与 -同一 harness 下其它条目的可比性。确实用了就在 `note` 里写清楚，否则不该上榜。） +MCP / 自定义系统提示——它们对产出的影响不亚于换 harness，这类 run 与同一 +harness 下其它条目不可比，**不予收录**。在 `note` 里写明也没用：榜单身份只看 +模型 × harness × effort，note 不另立身份，它仍会混进本不该可比的那一行。） ## Before you open the PR / 提交前 diff --git a/README.en.md b/README.en.md index 840ca53..d541116 100644 --- a/README.en.md +++ b/README.en.md @@ -21,7 +21,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x - **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination. - **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does. - The unit of evaluation here is therefore an **Agent**: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family). -- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. If you did use any, it is no longer the same agent; disclose it in the `note`. +- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. Such runs are **not accepted** (the leaderboard identity is just model × harness × effort, so a `note` does not give them a separate identity). ## Registry diff --git a/README.md b/README.md index e402ee9..16236d9 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ NAGI STUDIO 的 LLM 测评案例集：同一段提示词，不同「模型 × Ha - **模型（Model）**：权重与推理引擎本身，如 GPT-5.5、Gemini 3.1 Pro、Claude Fable 5。思考配额（effort）是模型自身的能力旋钮——同一份权重在高/低思考强度下的推理深度判若两个模型，因此它和权重同属「模型」这一维度，脱离 effort 谈模型表现并不公允；本仓库默认拉满（Max），并在每个组合里如实标注。 - **Harness（运行环境）**：包裹模型的产品/脚手架，如 Codex CLI、Cursor、AntiGravity、Claude 网页版——它决定工具调用、系统提示词、上下文管理与续写策略，对最终产出的影响往往不亚于模型本身。 - 因此本仓库的测评单位是一个 **Agent**：同一个模型（含其思考配额）换一个 Harness，就是另一个 Agent，记为不同条目（例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5（xhigh）是同一模型家族的两个 Agent）。 -- **测的是 Harness 开箱即用的默认形态**：请勿额外加载 Harness 自带以外的 skill / 插件 / MCP / 自定义系统提示——它们对产出的影响不亚于换 Harness，会破坏 Agent 之间的可比性。确实用了，就不是同一套 Agent，必须在 `note` 里如实说明。 +- **测的是 Harness 开箱即用的默认形态**：请勿额外加载 Harness 自带以外的 skill / 插件 / MCP / 自定义系统提示——它们对产出的影响不亚于换 Harness，会破坏 Agent 之间的可比性。这类 run **不予收录**（榜单身份只看 模型 × Harness × effort，在 `note` 里说明也不另立身份）。 ## 已测组合 / Registry