Frame the eval unit as an Agent (Model + Harness) in both READMEs

Lead the model-vs-harness section with the equation Model + Harness =
Agent (English equation + Chinese gloss): a model is just weights; wrapped
in a harness it becomes an agent that does the work, and that agent is the
unit this bench evaluates. Effort is reframed as a parameter of the agent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jesse ZhangandClaude Fable 5 committed 2026-06-11 15:22:17 +01:00
1 parent debe4105a8
commit 1857323b57
2 files changed
+6 -2

No files matched your search

+3 -1
View File
@@ -16,9 +16,11 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
## Models vs harnesses
> **Model + Harness = Agent** — a model is just weights; wrap it in a runtime and it becomes an agent that can actually do the work. That agent is the unit this bench evaluates.
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5.
- **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
- The unit of evaluation here is therefore the **combination**: the same model under a different harness or effort level is a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two records of the same model family). Effort defaults to max; new effort variants of an existing combination can be added later.
- The unit of evaluation here is therefore an **Agent**: the same model under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family). Effort is a parameter of that agent, defaulting to max.
## Registry
+3 -1
View File
@@ -16,9 +16,11 @@ NAGI STUDIO 的 LLM 测评案例集:同一段提示词,不同「模型 × Ha
## 模型与 Harness 的关系
> **Model + Harness = Agent** —— 模型只是权重,套上运行环境才成为一个能干活的智能体(Agent)。本仓库的测评单位就是一个 Agent。
- **模型(Model)**:权重与推理引擎本身,如 GPT-5.5、Gemini 3.1 Pro、Claude Fable 5。
- **Harness(运行环境)**:包裹模型的产品/脚手架,如 Codex CLI、Cursor、AntiGravity、Claude 网页版——它决定工具调用、系统提示词、上下文管理与续写策略,对最终产出的影响往往不亚于模型本身。
- 因此本仓库的测评单位是**组合**:同一个模型在不同 Harness 或不同思考配额下,记为不同条目(例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5(xhigh)是同一模型家族的两条记录)。思考配额默认拉满(Max),未来同一组合可以追加新配额版本。
- 因此本仓库的测评单位是一个 **Agent**:同一个模型换一个 Harness,就是另一个 Agent,记为不同条目(例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5(xhigh)是同一模型家族的两个 Agent)。思考配额(effort)是这个 Agent 的一个参数,默认拉满(Max)。
## 已测组合 / Registry