docs: reject runs with extra skills outright (addr Codex #25 P2)
A `note` disclosure does not separate a modified-harness run on the board: leaderboard identity is only model × harness × effort, so a skill-boosted run would still land in an Agent row it is explicitly not comparable with. Drop the "disclose in note" escape hatch and make it a flat non-acceptance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5cd30bac2b
commit
308c93ac81
3 files changed
+10
-8
No files matched your search
+1
-1
@@ -21,7 +21,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
|
||||
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination.
|
||||
- **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
|
||||
- The unit of evaluation here is therefore an **Agent**: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family).
|
||||
- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. If you did use any, it is no longer the same agent; disclose it in the `note`.
|
||||
- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. Such runs are **not accepted** (the leaderboard identity is just model × harness × effort, so a `note` does not give them a separate identity).
|
||||
|
||||
## Registry
|
||||
|
||||
|
||||
Reference in new issue
Block a user