Sort models by version desc and clarify effort belongs to the model (#18)
Reorder the Opus (4.8>4.7>4.6>4.5), MiMo (keep the two variants adjacent) and GLM (5.2>5.1>5 Turbo) entries so the registry sorts by version high-to-low instead of the previous scrambled order, adjusting each model JSON's `order` and regenerating the README registry tables. Also revise the model-vs-harness explainer: reasoning effort is a capability dial of the model itself (same weights at high vs low effort reason like different models), so it belongs to the "model" dimension alongside the weights rather than being framed as a side parameter of the agent. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jesse ZhangandClaude Opus 4.8 authored and GitHub committed
1 parent
3d5bf6967c
commit
7eaac09dc0
11 files changed
+21
-21
No files matched your search
+6
-6
@@ -18,9 +18,9 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
|
||||
|
||||
> **Model + Harness = Agent** — a model is just weights; wrap it in a runtime and it becomes an agent that can actually do the work. That agent is the unit this bench evaluates.
|
||||
|
||||
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5.
|
||||
- **Model**: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort is a capability dial of the model itself: the same weights at high vs low effort reason at depths that feel like two different models, so effort belongs to the "model" dimension alongside the weights — judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination.
|
||||
- **Harness**: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
|
||||
- The unit of evaluation here is therefore an **Agent**: the same model under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family). Effort is a parameter of that agent, defaulting to max.
|
||||
- The unit of evaluation here is therefore an **Agent**: the same model (including its effort) under a different harness is a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family).
|
||||
|
||||
## Registry
|
||||
|
||||
@@ -30,9 +30,9 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
|
||||
| Model | Vendor | Harness x Effort | Runs |
|
||||
|---|---|---|---|
|
||||
| Claude Fable 5 | Anthropic | Claude Web App · Max<br>Claude Code · Max<br>Cursor · High | 06 |
|
||||
| Claude Opus 4.6 | Anthropic | Claude Code · Max<br>Cursor · Max | 04 |
|
||||
| Claude Opus 4.7 | Anthropic | Claude Code · Max<br>Cursor · Max | 05 |
|
||||
| Claude Opus 4.8 | Anthropic | Claude Code · Max | 05 |
|
||||
| Claude Opus 4.7 | Anthropic | Claude Code · Max<br>Cursor · Max | 05 |
|
||||
| Claude Opus 4.6 | Anthropic | Claude Code · Max<br>Cursor · Max | 04 |
|
||||
| Claude Opus 4.5 | Anthropic | Cursor · Thinking | 02 |
|
||||
| Claude Sonnet 4.6 | Anthropic | Claude Code · Max<br>Cursor · High | 03 |
|
||||
| Claude Haiku 4.5 | Anthropic | Claude Code · Default | 02 |
|
||||
@@ -52,13 +52,13 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
|
||||
| Doubao Seed 2.0 Pro | ByteDance | Doubao Web · Pro Mode | 01 |
|
||||
| Doubao Seed 2.0 Mini | ByteDance | Doubao Web · Fast Mode | 01 |
|
||||
| MiMo v2.5 Pro | Xiaomi | Claude Code · Max | 02 |
|
||||
| Kimi K2.7-Code | Moonshot AI | Kimi Code · Thinking | 02 |
|
||||
| MiMo v2.5 Pro UltraSpeed | Xiaomi | Web · Default | 04 |
|
||||
| Kimi K2.7-Code | Moonshot AI | Kimi Code · Thinking | 02 |
|
||||
| Kimi K2.6 | Moonshot AI | Kimi Code · Thinking | 02 |
|
||||
| MiniMax M3 | MiniMax | MiniMax Code Web · Thinking | 02 |
|
||||
| MiniMax M2.7 | MiniMax | MiniMax Code Web · Thinking | 01 |
|
||||
| GLM-5 Turbo | Zhipu AI | ZCode · Thinking | 03 |
|
||||
| GLM-5.2 | Zhipu AI | ZCode · Max | 03 |
|
||||
| GLM-5 Turbo | Zhipu AI | ZCode · Thinking | 03 |
|
||||
| Qwen3.7-Max | Alibaba | Qoder · Default | 02 |
|
||||
| Step 3.7 Flash | StepFun | Claude Code · High | 02 |
|
||||
|
||||
|
||||
@@ -18,9 +18,9 @@ NAGI STUDIO 的 LLM 测评案例集：同一段提示词，不同「模型 × Ha
|
||||
|
||||
> **Model + Harness = Agent** —— 模型只是权重，套上运行环境才成为一个能干活的智能体（Agent）。本仓库的测评单位就是一个 Agent。
|
||||
|
||||
- **模型（Model）**：权重与推理引擎本身，如 GPT-5.5、Gemini 3.1 Pro、Claude Fable 5。
|
||||
- **模型（Model）**：权重与推理引擎本身，如 GPT-5.5、Gemini 3.1 Pro、Claude Fable 5。思考配额（effort）是模型自身的能力旋钮——同一份权重在高/低思考强度下的推理深度判若两个模型，因此它和权重同属「模型」这一维度，脱离 effort 谈模型表现并不公允；本仓库默认拉满（Max），并在每个组合里如实标注。
|
||||
- **Harness（运行环境）**：包裹模型的产品/脚手架，如 Codex CLI、Cursor、AntiGravity、Claude 网页版——它决定工具调用、系统提示词、上下文管理与续写策略，对最终产出的影响往往不亚于模型本身。
|
||||
- 因此本仓库的测评单位是一个 **Agent**：同一个模型换一个 Harness，就是另一个 Agent，记为不同条目（例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5（xhigh）是同一模型家族的两个 Agent）。思考配额（effort）是这个 Agent 的一个参数，默认拉满（Max）。
|
||||
- 因此本仓库的测评单位是一个 **Agent**：同一个模型（含其思考配额）换一个 Harness，就是另一个 Agent，记为不同条目（例如 GPT-5.5 Pro 与跑在 Codex CLI 里的 GPT-5.5（xhigh）是同一模型家族的两个 Agent）。
|
||||
|
||||
## 已测组合 / Registry
|
||||
|
||||
@@ -30,9 +30,9 @@ NAGI STUDIO 的 LLM 测评案例集：同一段提示词，不同「模型 × Ha
|
||||
| 模型 | 厂商 | 运行环境（Harness）× 思考配额 | 产出 |
|
||||
|---|---|---|---|
|
||||
| Claude Fable 5 | Anthropic | Claude Web App · Max<br>Claude Code · Max<br>Cursor · High | 06 |
|
||||
| Claude Opus 4.6 | Anthropic | Claude Code · Max<br>Cursor · Max | 04 |
|
||||
| Claude Opus 4.7 | Anthropic | Claude Code · Max<br>Cursor · Max | 05 |
|
||||
| Claude Opus 4.8 | Anthropic | Claude Code · Max | 05 |
|
||||
| Claude Opus 4.7 | Anthropic | Claude Code · Max<br>Cursor · Max | 05 |
|
||||
| Claude Opus 4.6 | Anthropic | Claude Code · Max<br>Cursor · Max | 04 |
|
||||
| Claude Opus 4.5 | Anthropic | Cursor · Thinking | 02 |
|
||||
| Claude Sonnet 4.6 | Anthropic | Claude Code · Max<br>Cursor · High | 03 |
|
||||
| Claude Haiku 4.5 | Anthropic | Claude Code · Default | 02 |
|
||||
@@ -52,13 +52,13 @@ NAGI STUDIO 的 LLM 测评案例集：同一段提示词，不同「模型 × Ha
|
||||
| Doubao Seed 2.0 Pro | ByteDance | Doubao Web · Pro Mode | 01 |
|
||||
| Doubao Seed 2.0 Mini | ByteDance | Doubao Web · Fast Mode | 01 |
|
||||
| MiMo v2.5 Pro | Xiaomi | Claude Code · Max | 02 |
|
||||
| Kimi K2.7-Code | Moonshot AI | Kimi Code · Thinking | 02 |
|
||||
| MiMo v2.5 Pro UltraSpeed | Xiaomi | Web · Default | 04 |
|
||||
| Kimi K2.7-Code | Moonshot AI | Kimi Code · Thinking | 02 |
|
||||
| Kimi K2.6 | Moonshot AI | Kimi Code · Thinking | 02 |
|
||||
| MiniMax M3 | MiniMax | MiniMax Code Web · Thinking | 02 |
|
||||
| MiniMax M2.7 | MiniMax | MiniMax Code Web · Thinking | 01 |
|
||||
| GLM-5 Turbo | Zhipu AI | ZCode · Thinking | 03 |
|
||||
| GLM-5.2 | Zhipu AI | ZCode · Max | 03 |
|
||||
| GLM-5 Turbo | Zhipu AI | ZCode · Thinking | 03 |
|
||||
| Qwen3.7-Max | Alibaba | Qoder · Default | 02 |
|
||||
| Step 3.7 Flash | StepFun | Claude Code · High | 02 |
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"harness": "Cursor",
|
||||
"effort": "Thinking",
|
||||
"artifactDir": "claude-opus-4-5/cursor-thinking",
|
||||
"order": 18,
|
||||
"order": 17,
|
||||
"runs": {
|
||||
"mythos-craft": {
|
||||
"note": {
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"harness": "Claude Code",
|
||||
"effort": "Max",
|
||||
"artifactDir": "claude-opus-4-6/claude-code-max",
|
||||
"order": 14.6,
|
||||
"order": 16,
|
||||
"runs": {
|
||||
"pelican-cycling": {
|
||||
"note": {
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"harness": "Cursor",
|
||||
"effort": "Max",
|
||||
"artifactDir": "claude-opus-4-6/cursor-max",
|
||||
"order": 17,
|
||||
"order": 16.5,
|
||||
"runs": {
|
||||
"mythos-craft": {
|
||||
"note": {
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"harness": "Claude Code",
|
||||
"effort": "Max",
|
||||
"artifactDir": "claude-opus-4-7/claude-code-max",
|
||||
"order": 14.7,
|
||||
"order": 15,
|
||||
"runs": {
|
||||
"mythos-craft": {
|
||||
"note": {
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"harness": "Cursor",
|
||||
"effort": "Max",
|
||||
"artifactDir": "claude-opus-4-7/cursor-max",
|
||||
"order": 16,
|
||||
"order": 15.5,
|
||||
"runs": {
|
||||
"mythos-craft": {
|
||||
"note": {
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"harness": "Claude Code",
|
||||
"effort": "Max",
|
||||
"artifactDir": "claude-opus-4-8/claude-code-max",
|
||||
"order": 15,
|
||||
"order": 14,
|
||||
"runs": {
|
||||
"mythos-craft": [
|
||||
{
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@
|
||||
"harness": "ZCode",
|
||||
"effort": "Max",
|
||||
"artifactDir": "glm-5-2/zcode-max",
|
||||
"order": 121,
|
||||
"order": 119,
|
||||
"runs": {
|
||||
"mythos-craft": {
|
||||
"note": {
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"harness": "ZCode",
|
||||
"effort": "Thinking",
|
||||
"artifactDir": "glm-5-turbo/zcode-thinking",
|
||||
"order": 119,
|
||||
"order": 121,
|
||||
"runs": {
|
||||
"mythos-craft": [
|
||||
{
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"vendor": "Xiaomi",
|
||||
"harness": "Web",
|
||||
"effort": "Default",
|
||||
"order": 99,
|
||||
"order": 97.5,
|
||||
"runs": {
|
||||
"mythos-craft": [
|
||||
{
|
||||
|
||||
Reference in new issue
Block a user