docs: require running the harness as it ships (no extra skills)

Address fairness concern from #24: the same harness can load completely
different skills / plugins / MCP / custom system prompts, which shifts the
output as much as swapping the harness and breaks Agent comparability.

- README.md / README.en.md: add a bullet to the Model + Harness section
  stating the harness is evaluated in its default, out-of-the-box form.
- AGENTS.md: add a "Run the harness as it ships" rule — no extra skills
  beyond what the harness bundles; disclose any deviation in `note`.

Enforcement stays honor-system + the PR generation screenshot.

Closes #24

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jesse ZhangandClaude Opus 4.8 committed 2026-06-24 20:18:08 +01:00
1 parent eaf2737ecc
commit 5cd30bac2b
3 files changed
+14

No files matched your search

+1
View File
@@ -21,6 +21,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination.
- **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
- The unit of evaluation here is therefore an **Agent**: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family).
- **The harness is evaluated as it ships, out of the box**: do not load skills / plugins / MCP servers / custom system prompts beyond what it bundles by default — they shift the output as much as swapping the harness does and break comparability between agents. If you did use any, it is no longer the same agent; disclose it in the `note`.
## Registry