diff --git a/README.en.md b/README.en.md index 8e06a6e..707adc9 100644 --- a/README.en.md +++ b/README.en.md @@ -18,9 +18,9 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x > **Model + Harness = Agent** — a model is just weights; wrap it in a runtime and it becomes an agent that can actually do the work. That agent is the unit this bench evaluates. -- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. +- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination. - **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does. -- The unit of evaluation here is therefore an **Agent**: the same model under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family). Effort is a parameter of that agent, defaulting to max. +- The unit of evaluation here is therefore an **Agent**: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family). ## Registry @@ -30,9 +30,9 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x | Model | Vendor | Harness x Effort | Runs | |---|---|---|---| | Claude Fable 5 | Anthropic | Claude Web App · Max
Claude Code · Max
Cursor · High | 06 | -| Claude Opus 4.6 | Anthropic | Claude Code · Max
Cursor · Max | 04 | -| Claude Opus 4.7 | Anthropic | Claude Code · Max
Cursor · Max | 05 | | Claude Opus 4.8 | Anthropic | Claude Code · Max | 05 | +| Claude Opus 4.7 | Anthropic | Claude Code · Max
Cursor · Max | 05 | +| Claude Opus 4.6 | Anthropic | Claude Code · Max
Cursor · Max | 04 | | Claude Opus 4.5 | Anthropic | Cursor · Thinking | 02 | | Claude Sonnet 4.6 | Anthropic | Claude Code · Max
Cursor · High | 03 | | Claude Haiku 4.5 | Anthropic | Claude Code · Default | 02 | @@ -52,13 +52,13 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x | Doubao Seed 2.0 Pro | ByteDance | Doubao Web · Pro Mode | 01 | | Doubao Seed 2.0 Mini | ByteDance | Doubao Web · Fast Mode | 01 | | MiMo v2.5 Pro | Xiaomi | Claude Code · Max | 02 | -| Kimi K2.7-Code | Moonshot AI | Kimi Code · Thinking | 02 | | MiMo v2.5 Pro UltraSpeed | Xiaomi | Web · Default | 04 | +| Kimi K2.7-Code | Moonshot AI | Kimi Code · Thinking | 02 | | Kimi K2.6 | Moonshot AI | Kimi Code · Thinking | 02 | | MiniMax M3 | MiniMax | MiniMax Code Web · Thinking | 02 | | MiniMax M2.7 | MiniMax | MiniMax Code Web · Thinking | 01 | -| GLM-5 Turbo | Zhipu AI | ZCode · Thinking | 03 | | GLM-5.2 | Zhipu AI | ZCode · Max | 03 | +| GLM-5 Turbo | Zhipu AI | ZCode · Thinking | 03 | | Qwen3.7-Max | Alibaba | Qoder · Default | 02 | | Step 3.7 Flash | StepFun | Claude Code · High | 02 | diff --git a/README.md b/README.md index a1fb2a7..403bbf1 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,9 @@ NAGI STUDIO 的 LLM 测评案例集:同一段提示词,不同「模型 × Ha > **Model + Harness = Agent** —— 模型只是权重,套上运行环境才成为一个能干活的智能体(Agent)。本仓库的测评单位就是一个 Agent。 -- **模型(Model)**:权重与推理引擎本身,如 GPT-5.5、Gemini 3.1 Pro、Claude Fable 5。 +- **模型(Model)**:权重与推理引擎本身,如 GPT-5.5、Gemini 3.1 Pro、Claude Fable 5。思考配额(effort)是模型自身的能力旋钮——同一份权重在高/低思考强度下的推理深度判若两个模型,因此它和权重同属「模型」这一维度,脱离 effort 谈模型表现并不公允;本仓库默认拉满(Max),并在每个组合里如实标注。 - **Harness(运行环境)**:包裹模型的产品/脚手架,如 Codex CLI、Cursor、AntiGravity、Claude 网页版——它决定工具调用、系统提示词、上下文管理与续写策略,对最终产出的影响往往不亚于模型本身。 -- 因此本仓库的测评单位是一个 **Agent**:同一个模型换一个 Harness,就是另一个 Agent,记为不同条目(例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5(xhigh)是同一模型家族的两个 Agent)。思考配额(effort)是这个 Agent 的一个参数,默认拉满(Max)。 +- 因此本仓库的测评单位是一个 **Agent**:同一个模型(含其思考配额)换一个 Harness,就是另一个 Agent,记为不同条目(例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5(xhigh)是同一模型家族的两个 Agent)。 ## 已测组合 / Registry @@ -30,9 +30,9 @@ NAGI STUDIO 的 LLM 测评案例集:同一段提示词,不同「模型 × Ha | 模型 | 厂商 | 运行环境(Harness)× 思考配额 | 产出 | |---|---|---|---| | Claude Fable 5 | Anthropic | Claude Web App · Max
Claude Code · Max
Cursor · High | 06 | -| Claude Opus 4.6 | Anthropic | Claude Code · Max
Cursor · Max | 04 | -| Claude Opus 4.7 | Anthropic | Claude Code · Max
Cursor · Max | 05 | | Claude Opus 4.8 | Anthropic | Claude Code · Max | 05 | +| Claude Opus 4.7 | Anthropic | Claude Code · Max
Cursor · Max | 05 | +| Claude Opus 4.6 | Anthropic | Claude Code · Max
Cursor · Max | 04 | | Claude Opus 4.5 | Anthropic | Cursor · Thinking | 02 | | Claude Sonnet 4.6 | Anthropic | Claude Code · Max
Cursor · High | 03 | | Claude Haiku 4.5 | Anthropic | Claude Code · Default | 02 | @@ -52,13 +52,13 @@ NAGI STUDIO 的 LLM 测评案例集:同一段提示词,不同「模型 × Ha | Doubao Seed 2.0 Pro | ByteDance | Doubao Web · Pro Mode | 01 | | Doubao Seed 2.0 Mini | ByteDance | Doubao Web · Fast Mode | 01 | | MiMo v2.5 Pro | Xiaomi | Claude Code · Max | 02 | -| Kimi K2.7-Code | Moonshot AI | Kimi Code · Thinking | 02 | | MiMo v2.5 Pro UltraSpeed | Xiaomi | Web · Default | 04 | +| Kimi K2.7-Code | Moonshot AI | Kimi Code · Thinking | 02 | | Kimi K2.6 | Moonshot AI | Kimi Code · Thinking | 02 | | MiniMax M3 | MiniMax | MiniMax Code Web · Thinking | 02 | | MiniMax M2.7 | MiniMax | MiniMax Code Web · Thinking | 01 | -| GLM-5 Turbo | Zhipu AI | ZCode · Thinking | 03 | | GLM-5.2 | Zhipu AI | ZCode · Max | 03 | +| GLM-5 Turbo | Zhipu AI | ZCode · Thinking | 03 | | Qwen3.7-Max | Alibaba | Qoder · Default | 02 | | Step 3.7 Flash | StepFun | Claude Code · High | 02 | diff --git a/models/claude-opus-4-5-cursor.json b/models/claude-opus-4-5-cursor.json index 6693af4..04821d2 100644 --- a/models/claude-opus-4-5-cursor.json +++ b/models/claude-opus-4-5-cursor.json @@ -4,7 +4,7 @@ "harness": "Cursor", "effort": "Thinking", "artifactDir": "claude-opus-4-5/cursor-thinking", - "order": 18, + "order": 17, "runs": { "mythos-craft": { "note": { diff --git a/models/claude-opus-4-6-cc.json b/models/claude-opus-4-6-cc.json index 7429f95..ae92474 100644 --- a/models/claude-opus-4-6-cc.json +++ b/models/claude-opus-4-6-cc.json @@ -4,7 +4,7 @@ "harness": "Claude Code", "effort": "Max", "artifactDir": "claude-opus-4-6/claude-code-max", - "order": 14.6, + "order": 16, "runs": { "pelican-cycling": { "note": { diff --git a/models/claude-opus-4-6-cursor.json b/models/claude-opus-4-6-cursor.json index d31ceaf..44527de 100644 --- a/models/claude-opus-4-6-cursor.json +++ b/models/claude-opus-4-6-cursor.json @@ -4,7 +4,7 @@ "harness": "Cursor", "effort": "Max", "artifactDir": "claude-opus-4-6/cursor-max", - "order": 17, + "order": 16.5, "runs": { "mythos-craft": { "note": { diff --git a/models/claude-opus-4-7-cc.json b/models/claude-opus-4-7-cc.json index e498150..406af91 100644 --- a/models/claude-opus-4-7-cc.json +++ b/models/claude-opus-4-7-cc.json @@ -4,7 +4,7 @@ "harness": "Claude Code", "effort": "Max", "artifactDir": "claude-opus-4-7/claude-code-max", - "order": 14.7, + "order": 15, "runs": { "mythos-craft": { "note": { diff --git a/models/claude-opus-4-7-cursor.json b/models/claude-opus-4-7-cursor.json index 3c2f86a..af9bc6f 100644 --- a/models/claude-opus-4-7-cursor.json +++ b/models/claude-opus-4-7-cursor.json @@ -4,7 +4,7 @@ "harness": "Cursor", "effort": "Max", "artifactDir": "claude-opus-4-7/cursor-max", - "order": 16, + "order": 15.5, "runs": { "mythos-craft": { "note": { diff --git a/models/claude-opus-4-8-cc.json b/models/claude-opus-4-8-cc.json index a04c49f..0228454 100644 --- a/models/claude-opus-4-8-cc.json +++ b/models/claude-opus-4-8-cc.json @@ -4,7 +4,7 @@ "harness": "Claude Code", "effort": "Max", "artifactDir": "claude-opus-4-8/claude-code-max", - "order": 15, + "order": 14, "runs": { "mythos-craft": [ { diff --git a/models/glm-5-2.json b/models/glm-5-2.json index cceb09e..4349192 100644 --- a/models/glm-5-2.json +++ b/models/glm-5-2.json @@ -4,7 +4,7 @@ "harness": "ZCode", "effort": "Max", "artifactDir": "glm-5-2/zcode-max", - "order": 121, + "order": 119, "runs": { "mythos-craft": { "note": { diff --git a/models/glm-5-turbo.json b/models/glm-5-turbo.json index bc67582..e131234 100644 --- a/models/glm-5-turbo.json +++ b/models/glm-5-turbo.json @@ -4,7 +4,7 @@ "harness": "ZCode", "effort": "Thinking", "artifactDir": "glm-5-turbo/zcode-thinking", - "order": 119, + "order": 121, "runs": { "mythos-craft": [ { diff --git a/models/mimo-v2-5-pro-ultraspeed.json b/models/mimo-v2-5-pro-ultraspeed.json index 824795e..154bf3c 100644 --- a/models/mimo-v2-5-pro-ultraspeed.json +++ b/models/mimo-v2-5-pro-ultraspeed.json @@ -3,7 +3,7 @@ "vendor": "Xiaomi", "harness": "Web", "effort": "Default", - "order": 99, + "order": 97.5, "runs": { "mythos-craft": [ {