docs: require running the harness as it ships (no extra skills)
Address fairness concern from #24: the same harness can load completely different skills / plugins / MCP / custom system prompts, which shifts the output as much as swapping the harness and breaks Agent comparability. - README.md / README.en.md: add a bullet to the Model + Harness section stating the harness is evaluated in its default, out-of-the-box form. - AGENTS.md: add a "Run the harness as it ships" rule — no extra skills beyond what the harness bundles; disclose any deviation in `note`. Enforcement stays honor-system + the PR generation screenshot. Closes #24 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
eaf2737ecc
commit
5cd30bac2b
3 files changed
+14
No files matched your search
@@ -21,6 +21,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
|
||||
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination.
|
||||
- **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
|
||||
- The unit of evaluation here is therefore an **Agent**: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family).
|
||||
- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. If you did use any, it is no longer the same agent; disclose it in the `note`.
|
||||
|
||||
## Registry
|
||||
|
||||
|
||||
Reference in new issue
Block a user