diff --git a/README.en.md b/README.en.md index 651a022..c3c413c 100644 --- a/README.en.md +++ b/README.en.md @@ -57,8 +57,8 @@ Pending: GLM-5.1 (PRs welcome) ## Structure ``` -outputs//. raw model artifacts (HTML / SVG), filename = case id -models/.json combination registry: label / vendor / harness / effort / order / run notes +outputs//. raw model artifacts (HTML / SVG), filename = case id +models/.json agent registry: label / vendor / harness / effort / artifactDir / order / run notes cases.json case definitions (bilingual prompts, maintainer-owned) scripts/validate-data.ts data validation (CI runs it on every PR) ``` @@ -75,16 +75,16 @@ bun scripts/validate-data.ts # the same data validation CI runs > **If you are an AI agent (Claude Code / Codex / Cursor, etc.), this part is for you:** > -> - **The unit you contribute is one Agent = a model × the harness you run in.** The `` (lowercase letters / digits / dashes) encodes that pairing — e.g. `gpt-5-5-pro`, `claude-fable-5-cc` (cc = Claude Code). The same model in a different harness is a different Agent with a different id. +> - **The unit you contribute is one Agent = a model × the harness you run in.** The `` (lowercase letters / digits / dashes) encodes that pairing — e.g. `gpt-5-5-pro`, `claude-fable-5-cc` (cc = Claude Code). The same model in a different harness is a different Agent with a different id. The `` is the vote and leaderboard identity; do not rename it just to reorganize files. > - **There are two cases right now**, defined in [`cases.json`](./cases.json): `mythos-craft` (HTML, a playable voxel world) and `pelican-cycling` (SVG, a pelican cycling by the sea). The prompt lives in `cases.json` — **use it verbatim, do not paraphrase**. > - **To find what's missing**: read `cases.json` (all cases) and `models/*.json` (what each Agent has run) — a `` absent from an Agent's json is an open slot; any row in the Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill both. > - **Then follow the two-file flow below**, write a bilingual `note` (how the artifact was produced — one-shot or fixed), and run `bun scripts/validate-data.ts` until it passes before opening a PR. Contributions are data-only — no code changes needed: -1. Artifact: `outputs//.` (e.g. `outputs/gpt-5-5-pro/pelican-cycling.svg`). -2. Registration: `models/.json` (new file for a new combination; add to `runs` for an existing one) — see the schema example in the Chinese README. +1. Artifact: preferably `outputs///.` (e.g. `outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg`). +2. Registration: `models/.json` (new file for a new combination; add to `runs` for an existing one) with `artifactDir` pointing at the artifact folder — see the schema example in the Chinese README. -CI-enforced rules: dash-only lowercase ids matching the `outputs/` folder name; a **required bilingual provenance `note`** per run (which harness, what effort, one-shot or fixed); every declared run must have its artifact; multiple versions of the same combination x case go in an array with distinct `file` names (e.g. `pelican-cycling-2.svg`). Set `contributor` to your GitHub username — the site shows your avatar linked to your profile next to the run. Fill in `harness` and `effort` truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via [lobe-icons](https://github.com/lobehub/lobe-icons)) — contributors never touch icons. +CI-enforced rules: dash-only lowercase `agent-id`s as stable identities; without `artifactDir`, artifacts default to `outputs//`; with `artifactDir`, use one or two lowercase dash-case path segments, preferably `/`. A bilingual provenance `note` is required per run (which harness, what effort, one-shot or fixed); every declared run must have its artifact; multiple versions of the same combination x case go in an array with distinct `file` names (e.g. `pelican-cycling-2.svg`). Set `contributor` to your GitHub username — the site shows your avatar linked to your profile next to the run. Fill in `harness` and `effort` truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via [lobe-icons](https://github.com/lobehub/lobe-icons)) — contributors never touch icons. PRs get data validation from CI; merges to `main` rebuild the site automatically (usually instant, at most 6 hours). diff --git a/README.md b/README.md index 8ac2fd2..d06c082 100644 --- a/README.md +++ b/README.md @@ -57,8 +57,8 @@ NAGI STUDIO 的 LLM 测评案例集:同一段提示词,不同「模型 × Ha ## 结构 / Structure ``` -outputs//. 模型产出原文件(HTML / SVG),文件名 = 案例 id -models/.json 组合登记:label / vendor / harness / effort / order / 各案例运行备注 +outputs//. 模型产出原文件(HTML / SVG),文件名 = 案例 id +models/.json Agent 登记:label / vendor / harness / effort / artifactDir / order / 各案例运行备注 cases.json 案例定义:双语标题与提示词(维护者维护) scripts/validate-data.ts 数据校验(CI 对每个 PR 自动执行) ``` @@ -75,15 +75,15 @@ bun scripts/validate-data.ts # 与 CI 相同的数据校验 > **如果你是一个 AI Agent(Claude Code / Codex / Cursor 等),这一段是给你看的:** > -> - **你贡献的单位是一个 Agent = 模型 × 你所在的 Harness。** ``(小写字母 / 数字 / 连字符)就编码这个组合,如 `gpt-5-5-pro`、`claude-fable-5-cc`(cc = Claude Code)。同一模型换个 Harness 就是另一个 Agent、另一个 id。 +> - **你贡献的单位是一个 Agent = 模型 × 你所在的 Harness。** ``(小写字母 / 数字 / 连字符)就编码这个组合,如 `gpt-5-5-pro`、`claude-fable-5-cc`(cc = Claude Code)。同一模型换个 Harness 就是另一个 Agent、另一个 id。`` 是投票与榜单身份,不要为了整理文件夹而改它。 > - **当前有两个案例**,定义在 [`cases.json`](./cases.json):`mythos-craft`(HTML,可玩体素世界)与 `pelican-cycling`(SVG,海边骑车的鹈鹕)。提示词以 `cases.json` 为准,**逐字使用、不要改写**。 > - **想知道还缺哪些**:读 `cases.json`(全部案例)与 `models/*.json`(每个 Agent 跑过的案例)——某个 Agent 的 json 里缺某个 `` 就是一个空位;上面 Registry 表里「产出」数小于案例总数的行,就是还缺案例的 Agent;全新 Agent 则两个案例都可补。 > - **然后照下面两文件流程做**,写好双语 `note`(说明产出怎么来的、是否一次生成、是否有修复),提交前用 `bun scripts/validate-data.ts` 自检通过再发 PR。 贡献**不需要改任何代码**,只涉及两类文件: -1. 产出文件:`outputs//.`(如 `outputs/gpt-5-5-pro/pelican-cycling.svg`)。 -2. 组合登记:`models/.json`(新组合新建文件;已有组合在 `runs` 里加一条): +1. 产出文件:推荐放在 `outputs///.`(如 `outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg`)。 +2. Agent 登记:`models/.json`(新组合新建文件;已有组合在 `runs` 里加一条),并用 `artifactDir` 指向产出目录: ```json { @@ -91,6 +91,7 @@ bun scripts/validate-data.ts # 与 CI 相同的数据校验 "vendor": "OpenAI", "harness": "ChatGPT Web", "effort": "Extended Pro", + "artifactDir": "gpt-5-5-pro/chatgpt-web-extended-pro", "order": 20, "runs": { "pelican-cycling": { @@ -105,7 +106,8 @@ bun scripts/validate-data.ts # 与 CI 相同的数据校验 ``` 规则(CI 自动校验,不满足会挂): -- `model-id` 只用小写字母/数字/连字符(如 `doubao-seed-2-0-pro`),文件名与 `outputs/` 目录名一致; +- `agent-id` 只用小写字母/数字/连字符(如 `doubao-seed-2-0-pro`),它是稳定身份;没有 `artifactDir` 时默认读取 `outputs//`; +- `artifactDir` 只用一层或两层小写 dash-case 路径(推荐 `/`),它只是产出文件位置,不影响投票与榜单身份; - 每条 run 的 `note` **双语必填**——这是本仓库的可信度来源,必须写明产出怎么来的; - 声明的 run 必须有对应的产出文件; - 同一组合对同一案例可提交多个版本:`runs.` 写成数组,第二个版本起必须用 `file` 指定不同文件名(如 `pelican-cycling-2.svg`); diff --git a/models/claude-fable-5-cc.json b/models/claude-fable-5-cc.json index 74ef28d..8e88db9 100644 --- a/models/claude-fable-5-cc.json +++ b/models/claude-fable-5-cc.json @@ -3,6 +3,7 @@ "vendor": "Anthropic", "harness": "Claude Code", "effort": "Max", + "artifactDir": "claude-fable-5/claude-code-max", "order": 12, "runs": { "mythos-craft": { diff --git a/models/claude-fable-5-cursor.json b/models/claude-fable-5-cursor.json index a684e9a..2d44199 100644 --- a/models/claude-fable-5-cursor.json +++ b/models/claude-fable-5-cursor.json @@ -3,6 +3,7 @@ "vendor": "Anthropic", "harness": "Cursor", "effort": "High", + "artifactDir": "claude-fable-5/cursor-high", "order": 13, "runs": { "mythos-craft": { diff --git a/models/claude-fable-5.json b/models/claude-fable-5.json index 746af19..00105d2 100644 --- a/models/claude-fable-5.json +++ b/models/claude-fable-5.json @@ -3,6 +3,7 @@ "vendor": "Anthropic", "harness": "Claude Web App", "effort": "Max", + "artifactDir": "claude-fable-5/claude-web-app-max", "order": 10, "runs": { "mythos-craft": { diff --git a/models/claude-opus-4-8-cc.json b/models/claude-opus-4-8-cc.json index 2daa7ee..1f41c8b 100644 --- a/models/claude-opus-4-8-cc.json +++ b/models/claude-opus-4-8-cc.json @@ -1,6 +1,9 @@ { "label": "Claude Opus 4.8", "vendor": "Anthropic", + "harness": "Claude Code", + "effort": "Max", + "artifactDir": "claude-opus-4-8/claude-code-max", "order": 15, "runs": { "mythos-craft": [ @@ -37,7 +40,5 @@ "contributor": "OHMFOHS" } ] - }, - "harness": "Claude Code", - "effort": "Max" + } } diff --git a/models/composer-2-5-fast.json b/models/composer-2-5-fast.json index 536b7fe..c3468ae 100644 --- a/models/composer-2-5-fast.json +++ b/models/composer-2-5-fast.json @@ -3,6 +3,7 @@ "vendor": "Cursor", "harness": "Cursor", "effort": "Default", + "artifactDir": "composer-2-5/cursor-default", "order": 71, "runs": { "mythos-craft": { diff --git a/models/composer-2-5-grok-build.json b/models/composer-2-5-grok-build.json index e69ade3..a14de87 100644 --- a/models/composer-2-5-grok-build.json +++ b/models/composer-2-5-grok-build.json @@ -3,6 +3,7 @@ "vendor": "Cursor", "harness": "Grok Build TUI", "effort": "Default", + "artifactDir": "composer-2-5/grok-build-tui-default", "order": 72, "runs": { "mythos-craft": { diff --git a/models/composer-2-5.json b/models/composer-2-5.json index d267369..a722282 100644 --- a/models/composer-2-5.json +++ b/models/composer-2-5.json @@ -3,6 +3,7 @@ "vendor": "Cursor", "harness": "Cursor", "effort": "Max", + "artifactDir": "composer-2-5/cursor-max", "order": 70, "runs": { "mythos-craft": { diff --git a/models/deepseek-v4-pro.json b/models/deepseek-v4-pro.json index ef6887d..0b20c11 100644 --- a/models/deepseek-v4-pro.json +++ b/models/deepseek-v4-pro.json @@ -3,6 +3,7 @@ "vendor": "DeepSeek", "harness": "Claude Code", "effort": "Max", + "artifactDir": "deepseek-v4-pro/claude-code-max", "order": 90, "runs": { "mythos-craft": { diff --git a/models/doubao-seed-2-0-mini.json b/models/doubao-seed-2-0-mini.json index a24e440..f5278f2 100644 --- a/models/doubao-seed-2-0-mini.json +++ b/models/doubao-seed-2-0-mini.json @@ -3,6 +3,7 @@ "vendor": "ByteDance", "harness": "Doubao Web", "effort": "Fast Mode", + "artifactDir": "doubao-seed-2-0-mini/doubao-web-fast-mode", "order": 96, "runs": { "pelican-cycling": { diff --git a/models/doubao-seed-2-0-pro.json b/models/doubao-seed-2-0-pro.json index 1cdfc6d..6bd7db8 100644 --- a/models/doubao-seed-2-0-pro.json +++ b/models/doubao-seed-2-0-pro.json @@ -3,6 +3,7 @@ "vendor": "ByteDance", "harness": "Doubao Web", "effort": "Pro Mode", + "artifactDir": "doubao-seed-2-0-pro/doubao-web-pro-mode", "order": 95, "runs": { "pelican-cycling": { diff --git a/models/gemini-3-1-pro-aistudio.json b/models/gemini-3-1-pro-aistudio.json index 33a3e78..f002223 100644 --- a/models/gemini-3-1-pro-aistudio.json +++ b/models/gemini-3-1-pro-aistudio.json @@ -3,6 +3,7 @@ "vendor": "Google", "harness": "Google AI Studio", "effort": "High", + "artifactDir": "gemini-3-1-pro/google-ai-studio-high", "order": 42, "runs": { "pelican-cycling": { diff --git a/models/gemini-3-1-pro.json b/models/gemini-3-1-pro.json index c891628..e6a367e 100644 --- a/models/gemini-3-1-pro.json +++ b/models/gemini-3-1-pro.json @@ -3,6 +3,7 @@ "vendor": "Google", "harness": "AntiGravity", "effort": "High", + "artifactDir": "gemini-3-1-pro/antigravity-high", "order": 40, "runs": { "mythos-craft": { diff --git a/models/gemini-3-5-flash-aistudio.json b/models/gemini-3-5-flash-aistudio.json index 31af949..a12feb9 100644 --- a/models/gemini-3-5-flash-aistudio.json +++ b/models/gemini-3-5-flash-aistudio.json @@ -3,6 +3,7 @@ "vendor": "Google", "harness": "Google AI Studio", "effort": "High", + "artifactDir": "gemini-3-5-flash/google-ai-studio-high", "order": 52, "runs": { "pelican-cycling": { diff --git a/models/gemini-3-5-flash.json b/models/gemini-3-5-flash.json index 934fff2..3a0fdf9 100644 --- a/models/gemini-3-5-flash.json +++ b/models/gemini-3-5-flash.json @@ -3,6 +3,7 @@ "vendor": "Google", "harness": "AntiGravity", "effort": "High", + "artifactDir": "gemini-3-5-flash/antigravity-high", "order": 50, "runs": { "mythos-craft": { diff --git a/models/glm-5-1.json b/models/glm-5-1.json index ffea8ae..0ea5be1 100644 --- a/models/glm-5-1.json +++ b/models/glm-5-1.json @@ -1,5 +1,6 @@ { "label": "GLM-5.1", "vendor": "Zhipu AI", + "artifactDir": "glm-5-1/default", "order": 120 } diff --git a/models/gpt-5-5-pro.json b/models/gpt-5-5-pro.json index 64373ec..bcb701c 100644 --- a/models/gpt-5-5-pro.json +++ b/models/gpt-5-5-pro.json @@ -3,6 +3,7 @@ "vendor": "OpenAI", "harness": "ChatGPT Web", "effort": "Extended Pro", + "artifactDir": "gpt-5-5-pro/chatgpt-web-extended-pro", "order": 20, "runs": { "mythos-craft": { diff --git a/models/gpt-5-5-xhigh.json b/models/gpt-5-5-xhigh.json index 6543216..cc2e161 100644 --- a/models/gpt-5-5-xhigh.json +++ b/models/gpt-5-5-xhigh.json @@ -3,6 +3,7 @@ "vendor": "OpenAI", "harness": "Codex CLI", "effort": "xhigh", + "artifactDir": "gpt-5-5/codex-cli-xhigh", "order": 30, "runs": { "mythos-craft": { diff --git a/models/grok-build.json b/models/grok-build.json index 84ba41a..cc0cbc8 100644 --- a/models/grok-build.json +++ b/models/grok-build.json @@ -3,6 +3,7 @@ "vendor": "xAI", "harness": "Grok Build TUI", "effort": "Max", + "artifactDir": "grok-build/grok-build-tui-max", "order": 60, "runs": { "mythos-craft": { diff --git a/models/kimi-k2-6.json b/models/kimi-k2-6.json index ad0ce1f..fc979ad 100644 --- a/models/kimi-k2-6.json +++ b/models/kimi-k2-6.json @@ -1,9 +1,10 @@ { "label": "Kimi K2.6", "vendor": "Moonshot AI", - "order": 100, "harness": "Kimi Code", "effort": "Thinking", + "artifactDir": "kimi-k2-6/kimi-code-thinking", + "order": 100, "runs": { "mythos-craft": { "note": { diff --git a/models/kimi-k2-7-code.json b/models/kimi-k2-7-code.json index 94433bf..80e773f 100644 --- a/models/kimi-k2-7-code.json +++ b/models/kimi-k2-7-code.json @@ -1,9 +1,10 @@ { "label": "Kimi K2.7-Code", "vendor": "Moonshot AI", - "order": 98, "harness": "Kimi Code", "effort": "Thinking", + "artifactDir": "kimi-k2-7-code/kimi-code-thinking", + "order": 98, "runs": { "pelican-cycling": { "note": { diff --git a/models/minimax-m3.json b/models/minimax-m3.json index 6c93e9e..dc19973 100644 --- a/models/minimax-m3.json +++ b/models/minimax-m3.json @@ -1,9 +1,10 @@ { "label": "MiniMax M3", "vendor": "MiniMax", - "order": 110, "harness": "MiniMax Code Web", "effort": "Thinking", + "artifactDir": "minimax-m3/minimax-code-web-thinking", + "order": 110, "runs": { "pelican-cycling": { "note": { diff --git a/models/mistral-medium-3-5.json b/models/mistral-medium-3-5.json index 6ee7699..37cc365 100644 --- a/models/mistral-medium-3-5.json +++ b/models/mistral-medium-3-5.json @@ -3,6 +3,7 @@ "vendor": "Mistral AI", "harness": "Vibe", "effort": "Thinking", + "artifactDir": "mistral-medium-3-5/vibe-thinking", "order": 80, "runs": { "mythos-craft": { diff --git a/models/nex-n2-pro-cc.json b/models/nex-n2-pro-cc.json index f99bbd6..0fe7477 100644 --- a/models/nex-n2-pro-cc.json +++ b/models/nex-n2-pro-cc.json @@ -3,6 +3,7 @@ "vendor": "Nex-AGI", "harness": "Claude Code", "effort": "Max", + "artifactDir": "nex-n2-pro/claude-code-max", "order": 60, "runs": { "mythos-craft": { diff --git a/models/qwen3-7-max.json b/models/qwen3-7-max.json index 4640eef..3c54d66 100644 --- a/models/qwen3-7-max.json +++ b/models/qwen3-7-max.json @@ -1,6 +1,9 @@ { "label": "Qwen3.7-Max", "vendor": "Alibaba", + "harness": "Qoder", + "effort": "Default", + "artifactDir": "qwen3-7-max/qoder-default", "order": 130, "runs": { "mythos-craft": { @@ -17,7 +20,5 @@ }, "contributor": "Nagi-ovo" } - }, - "harness": "Qoder", - "effort": "Default" + } } diff --git a/models/xiaomi-mimo-v2-5-pro.json b/models/xiaomi-mimo-v2-5-pro.json index 9db56c1..cf63d66 100644 --- a/models/xiaomi-mimo-v2-5-pro.json +++ b/models/xiaomi-mimo-v2-5-pro.json @@ -3,6 +3,7 @@ "vendor": "Xiaomi", "harness": "Claude Code", "effort": "Max", + "artifactDir": "mimo-v2-5-pro/claude-code-max", "order": 97, "runs": { "mythos-craft": { diff --git a/outputs/claude-fable-5-cc/mythos-craft.html b/outputs/claude-fable-5/claude-code-max/mythos-craft.html similarity index 100% rename from outputs/claude-fable-5-cc/mythos-craft.html rename to outputs/claude-fable-5/claude-code-max/mythos-craft.html diff --git a/outputs/claude-fable-5-cc/pelican-cycling.svg b/outputs/claude-fable-5/claude-code-max/pelican-cycling.svg similarity index 100% rename from outputs/claude-fable-5-cc/pelican-cycling.svg rename to outputs/claude-fable-5/claude-code-max/pelican-cycling.svg diff --git a/outputs/claude-fable-5/mythos-craft.html b/outputs/claude-fable-5/claude-web-app-max/mythos-craft.html similarity index 100% rename from outputs/claude-fable-5/mythos-craft.html rename to outputs/claude-fable-5/claude-web-app-max/mythos-craft.html diff --git a/outputs/claude-fable-5/pelican-cycling.svg b/outputs/claude-fable-5/claude-web-app-max/pelican-cycling.svg similarity index 100% rename from outputs/claude-fable-5/pelican-cycling.svg rename to outputs/claude-fable-5/claude-web-app-max/pelican-cycling.svg diff --git a/outputs/claude-fable-5-cursor/mythos-craft.html b/outputs/claude-fable-5/cursor-high/mythos-craft.html similarity index 100% rename from outputs/claude-fable-5-cursor/mythos-craft.html rename to outputs/claude-fable-5/cursor-high/mythos-craft.html diff --git a/outputs/claude-fable-5-cursor/pelican-cycling.svg b/outputs/claude-fable-5/cursor-high/pelican-cycling.svg similarity index 100% rename from outputs/claude-fable-5-cursor/pelican-cycling.svg rename to outputs/claude-fable-5/cursor-high/pelican-cycling.svg diff --git a/outputs/claude-opus-4-8-cc/mythos-craft-2.html b/outputs/claude-opus-4-8/claude-code-max/mythos-craft-2.html similarity index 100% rename from outputs/claude-opus-4-8-cc/mythos-craft-2.html rename to outputs/claude-opus-4-8/claude-code-max/mythos-craft-2.html diff --git a/outputs/claude-opus-4-8-cc/mythos-craft.html b/outputs/claude-opus-4-8/claude-code-max/mythos-craft.html similarity index 100% rename from outputs/claude-opus-4-8-cc/mythos-craft.html rename to outputs/claude-opus-4-8/claude-code-max/mythos-craft.html diff --git a/outputs/claude-opus-4-8-cc/pelican-cycling-2.svg b/outputs/claude-opus-4-8/claude-code-max/pelican-cycling-2.svg similarity index 100% rename from outputs/claude-opus-4-8-cc/pelican-cycling-2.svg rename to outputs/claude-opus-4-8/claude-code-max/pelican-cycling-2.svg diff --git a/outputs/claude-opus-4-8-cc/pelican-cycling.svg b/outputs/claude-opus-4-8/claude-code-max/pelican-cycling.svg similarity index 100% rename from outputs/claude-opus-4-8-cc/pelican-cycling.svg rename to outputs/claude-opus-4-8/claude-code-max/pelican-cycling.svg diff --git a/outputs/composer-2-5-fast/mythos-craft.html b/outputs/composer-2-5/cursor-default/mythos-craft.html similarity index 100% rename from outputs/composer-2-5-fast/mythos-craft.html rename to outputs/composer-2-5/cursor-default/mythos-craft.html diff --git a/outputs/composer-2-5-fast/pelican-cycling.svg b/outputs/composer-2-5/cursor-default/pelican-cycling.svg similarity index 100% rename from outputs/composer-2-5-fast/pelican-cycling.svg rename to outputs/composer-2-5/cursor-default/pelican-cycling.svg diff --git a/outputs/composer-2-5/mythos-craft.html b/outputs/composer-2-5/cursor-max/mythos-craft.html similarity index 100% rename from outputs/composer-2-5/mythos-craft.html rename to outputs/composer-2-5/cursor-max/mythos-craft.html diff --git a/outputs/composer-2-5-grok-build/mythos-craft.html b/outputs/composer-2-5/grok-build-tui-default/mythos-craft.html similarity index 100% rename from outputs/composer-2-5-grok-build/mythos-craft.html rename to outputs/composer-2-5/grok-build-tui-default/mythos-craft.html diff --git a/outputs/composer-2-5-grok-build/pelican-cycling.svg b/outputs/composer-2-5/grok-build-tui-default/pelican-cycling.svg similarity index 100% rename from outputs/composer-2-5-grok-build/pelican-cycling.svg rename to outputs/composer-2-5/grok-build-tui-default/pelican-cycling.svg diff --git a/outputs/deepseek-v4-pro/mythos-craft.html b/outputs/deepseek-v4-pro/claude-code-max/mythos-craft.html similarity index 100% rename from outputs/deepseek-v4-pro/mythos-craft.html rename to outputs/deepseek-v4-pro/claude-code-max/mythos-craft.html diff --git a/outputs/deepseek-v4-pro/pelican-cycling.svg b/outputs/deepseek-v4-pro/claude-code-max/pelican-cycling.svg similarity index 100% rename from outputs/deepseek-v4-pro/pelican-cycling.svg rename to outputs/deepseek-v4-pro/claude-code-max/pelican-cycling.svg diff --git a/outputs/doubao-seed-2-0-mini/pelican-cycling.svg b/outputs/doubao-seed-2-0-mini/doubao-web-fast-mode/pelican-cycling.svg similarity index 100% rename from outputs/doubao-seed-2-0-mini/pelican-cycling.svg rename to outputs/doubao-seed-2-0-mini/doubao-web-fast-mode/pelican-cycling.svg diff --git a/outputs/doubao-seed-2-0-pro/pelican-cycling.svg b/outputs/doubao-seed-2-0-pro/doubao-web-pro-mode/pelican-cycling.svg similarity index 100% rename from outputs/doubao-seed-2-0-pro/pelican-cycling.svg rename to outputs/doubao-seed-2-0-pro/doubao-web-pro-mode/pelican-cycling.svg diff --git a/outputs/gemini-3-1-pro/mythos-craft.html b/outputs/gemini-3-1-pro/antigravity-high/mythos-craft.html similarity index 100% rename from outputs/gemini-3-1-pro/mythos-craft.html rename to outputs/gemini-3-1-pro/antigravity-high/mythos-craft.html diff --git a/outputs/gemini-3-1-pro/pelican-cycling-2.svg b/outputs/gemini-3-1-pro/antigravity-high/pelican-cycling-2.svg similarity index 100% rename from outputs/gemini-3-1-pro/pelican-cycling-2.svg rename to outputs/gemini-3-1-pro/antigravity-high/pelican-cycling-2.svg diff --git a/outputs/gemini-3-1-pro-aistudio/pelican-cycling.svg b/outputs/gemini-3-1-pro/google-ai-studio-high/pelican-cycling.svg similarity index 100% rename from outputs/gemini-3-1-pro-aistudio/pelican-cycling.svg rename to outputs/gemini-3-1-pro/google-ai-studio-high/pelican-cycling.svg diff --git a/outputs/gemini-3-5-flash/mythos-craft.html b/outputs/gemini-3-5-flash/antigravity-high/mythos-craft.html similarity index 100% rename from outputs/gemini-3-5-flash/mythos-craft.html rename to outputs/gemini-3-5-flash/antigravity-high/mythos-craft.html diff --git a/outputs/gemini-3-5-flash/pelican-cycling-2.svg b/outputs/gemini-3-5-flash/antigravity-high/pelican-cycling-2.svg similarity index 100% rename from outputs/gemini-3-5-flash/pelican-cycling-2.svg rename to outputs/gemini-3-5-flash/antigravity-high/pelican-cycling-2.svg diff --git a/outputs/gemini-3-5-flash-aistudio/pelican-cycling.svg b/outputs/gemini-3-5-flash/google-ai-studio-high/pelican-cycling.svg similarity index 100% rename from outputs/gemini-3-5-flash-aistudio/pelican-cycling.svg rename to outputs/gemini-3-5-flash/google-ai-studio-high/pelican-cycling.svg diff --git a/outputs/gpt-5-5-pro/mythos-craft.html b/outputs/gpt-5-5-pro/chatgpt-web-extended-pro/mythos-craft.html similarity index 100% rename from outputs/gpt-5-5-pro/mythos-craft.html rename to outputs/gpt-5-5-pro/chatgpt-web-extended-pro/mythos-craft.html diff --git a/outputs/gpt-5-5-pro/pelican-cycling.svg b/outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg similarity index 100% rename from outputs/gpt-5-5-pro/pelican-cycling.svg rename to outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg diff --git a/outputs/gpt-5-5-xhigh/mythos-craft.html b/outputs/gpt-5-5/codex-cli-xhigh/mythos-craft.html similarity index 100% rename from outputs/gpt-5-5-xhigh/mythos-craft.html rename to outputs/gpt-5-5/codex-cli-xhigh/mythos-craft.html diff --git a/outputs/gpt-5-5-xhigh/pelican-cycling.svg b/outputs/gpt-5-5/codex-cli-xhigh/pelican-cycling.svg similarity index 100% rename from outputs/gpt-5-5-xhigh/pelican-cycling.svg rename to outputs/gpt-5-5/codex-cli-xhigh/pelican-cycling.svg diff --git a/outputs/grok-build/mythos-craft.html b/outputs/grok-build/grok-build-tui-max/mythos-craft.html similarity index 100% rename from outputs/grok-build/mythos-craft.html rename to outputs/grok-build/grok-build-tui-max/mythos-craft.html diff --git a/outputs/grok-build/pelican-cycling.svg b/outputs/grok-build/grok-build-tui-max/pelican-cycling.svg similarity index 100% rename from outputs/grok-build/pelican-cycling.svg rename to outputs/grok-build/grok-build-tui-max/pelican-cycling.svg diff --git a/outputs/kimi-k2-6/mythos-craft.html b/outputs/kimi-k2-6/kimi-code-thinking/mythos-craft.html similarity index 100% rename from outputs/kimi-k2-6/mythos-craft.html rename to outputs/kimi-k2-6/kimi-code-thinking/mythos-craft.html diff --git a/outputs/kimi-k2-6/pelican-cycling.svg b/outputs/kimi-k2-6/kimi-code-thinking/pelican-cycling.svg similarity index 100% rename from outputs/kimi-k2-6/pelican-cycling.svg rename to outputs/kimi-k2-6/kimi-code-thinking/pelican-cycling.svg diff --git a/outputs/kimi-k2-7-code/pelican-cycling.svg b/outputs/kimi-k2-7-code/kimi-code-thinking/pelican-cycling.svg similarity index 100% rename from outputs/kimi-k2-7-code/pelican-cycling.svg rename to outputs/kimi-k2-7-code/kimi-code-thinking/pelican-cycling.svg diff --git a/outputs/xiaomi-mimo-v2-5-pro/mythos-craft.html b/outputs/mimo-v2-5-pro/claude-code-max/mythos-craft.html similarity index 100% rename from outputs/xiaomi-mimo-v2-5-pro/mythos-craft.html rename to outputs/mimo-v2-5-pro/claude-code-max/mythos-craft.html diff --git a/outputs/xiaomi-mimo-v2-5-pro/pelican-cycling.svg b/outputs/mimo-v2-5-pro/claude-code-max/pelican-cycling.svg similarity index 100% rename from outputs/xiaomi-mimo-v2-5-pro/pelican-cycling.svg rename to outputs/mimo-v2-5-pro/claude-code-max/pelican-cycling.svg diff --git a/outputs/minimax-m3/pelican-cycling.svg b/outputs/minimax-m3/minimax-code-web-thinking/pelican-cycling.svg similarity index 100% rename from outputs/minimax-m3/pelican-cycling.svg rename to outputs/minimax-m3/minimax-code-web-thinking/pelican-cycling.svg diff --git a/outputs/mistral-medium-3-5/mythos-craft.html b/outputs/mistral-medium-3-5/vibe-thinking/mythos-craft.html similarity index 100% rename from outputs/mistral-medium-3-5/mythos-craft.html rename to outputs/mistral-medium-3-5/vibe-thinking/mythos-craft.html diff --git a/outputs/mistral-medium-3-5/pelican-cycling.svg b/outputs/mistral-medium-3-5/vibe-thinking/pelican-cycling.svg similarity index 100% rename from outputs/mistral-medium-3-5/pelican-cycling.svg rename to outputs/mistral-medium-3-5/vibe-thinking/pelican-cycling.svg diff --git a/outputs/nex-n2-pro-cc/mythos-craft.html b/outputs/nex-n2-pro/claude-code-max/mythos-craft.html similarity index 100% rename from outputs/nex-n2-pro-cc/mythos-craft.html rename to outputs/nex-n2-pro/claude-code-max/mythos-craft.html diff --git a/outputs/nex-n2-pro-cc/pelican-cycling.svg b/outputs/nex-n2-pro/claude-code-max/pelican-cycling.svg similarity index 100% rename from outputs/nex-n2-pro-cc/pelican-cycling.svg rename to outputs/nex-n2-pro/claude-code-max/pelican-cycling.svg diff --git a/outputs/qwen3-7-max/mythos-craft.html b/outputs/qwen3-7-max/qoder-default/mythos-craft.html similarity index 100% rename from outputs/qwen3-7-max/mythos-craft.html rename to outputs/qwen3-7-max/qoder-default/mythos-craft.html diff --git a/outputs/qwen3-7-max/pelican-cycling.svg b/outputs/qwen3-7-max/qoder-default/pelican-cycling.svg similarity index 100% rename from outputs/qwen3-7-max/pelican-cycling.svg rename to outputs/qwen3-7-max/qoder-default/pelican-cycling.svg diff --git a/scripts/validate-data.ts b/scripts/validate-data.ts index 42552ba..dc53427 100644 --- a/scripts/validate-data.ts +++ b/scripts/validate-data.ts @@ -16,6 +16,7 @@ const cases = JSON.parse(readFileSync(join(root, 'cases.json'), 'utf8')) as Arra const caseKind = new Map(cases.map((c) => [c.id, c.kind])) const ID_RE = /^[a-z0-9]+(-[a-z0-9]+)*$/ +const ARTIFACT_DIR_RE = /^[a-z0-9]+(-[a-z0-9]+)*(\/[a-z0-9]+(-[a-z0-9]+)*)?$/ const errors: string[] = [] const isBilingual = (v: unknown): boolean => { @@ -25,15 +26,23 @@ const isBilingual = (v: unknown): boolean => { } const modelIds = new Set() +const artifactDirs = new Map() +const declaredArtifacts = new Set() for (const f of readdirSync(modelsDir).filter((f) => f.endsWith('.json'))) { const id = f.replace(/\.json$/, '') modelIds.add(id) if (!ID_RE.test(id)) { - errors.push(`models/${f}: id "${id}" must be lowercase letters/digits/dashes only (it doubles as the outputs/ folder name)`) + errors.push(`models/${f}: id "${id}" must be lowercase letters/digits/dashes only (it is the stable voting key and default outputs folder)`) } type RunJson = { note?: unknown; file?: unknown; contributor?: unknown } - let def: { label?: unknown; vendor?: unknown; order?: unknown; runs?: Record } + let def: { + label?: unknown + vendor?: unknown + artifactDir?: unknown + order?: unknown + runs?: Record + } try { def = JSON.parse(readFileSync(join(modelsDir, f), 'utf8')) } catch (e) { @@ -44,6 +53,19 @@ for (const f of readdirSync(modelsDir).filter((f) => f.endsWith('.json'))) { if (typeof def.label !== 'string' || !def.label.trim()) errors.push(`models/${f}: "label" is required`) if (typeof def.vendor !== 'string' || !def.vendor.trim()) errors.push(`models/${f}: "vendor" is required`) if (def.order !== undefined && typeof def.order !== 'number') errors.push(`models/${f}: "order" must be a number`) + if (def.artifactDir !== undefined && typeof def.artifactDir !== 'string') { + errors.push(`models/${f}: "artifactDir" must be a string`) + } + const artifactDir = typeof def.artifactDir === 'string' && def.artifactDir.trim() ? def.artifactDir.trim() : id + if (!ARTIFACT_DIR_RE.test(artifactDir)) { + errors.push(`models/${f}: artifactDir "${artifactDir}" must be one or two lowercase dash-case path segments, e.g. "claude-fable-5/claude-code-max"`) + } + const dirOwner = artifactDirs.get(artifactDir) + if (dirOwner) { + errors.push(`models/${f}: artifactDir "${artifactDir}" is already used by models/${dirOwner}.json`) + } else { + artifactDirs.set(artifactDir, id) + } for (const [caseId, runRaw] of Object.entries(def.runs ?? {})) { if (!caseKind.has(caseId)) { @@ -61,6 +83,12 @@ for (const f of readdirSync(modelsDir).filter((f) => f.endsWith('.json'))) { if (run?.file !== undefined && typeof run.file !== 'string') { errors.push(`models/${f}: ${where}.file must be a string`) } + if ( + typeof run?.file === 'string' && + (run.file.includes('/') || run.file.includes('\\') || run.file.includes('..')) + ) { + errors.push(`models/${f}: ${where}.file must be a filename, not a path`) + } if (run?.contributor !== undefined && (typeof run.contributor !== 'string' || !run.contributor.trim())) { errors.push(`models/${f}: ${where}.contributor must be a non-empty GitHub username`) } @@ -69,39 +97,31 @@ for (const f of readdirSync(modelsDir).filter((f) => f.endsWith('.json'))) { errors.push(`models/${f}: ${where} resolves to the same artifact "${file}" as another variant — extra variants must set a distinct "file"`) } seenFiles.add(file) - if (!existsSync(join(outputsDir, id, file))) { - errors.push(`models/${f}: ${where} declared but artifact missing: outputs/${id}/${file}`) + const artifact = `${artifactDir}/${file}` + declaredArtifacts.add(artifact) + if (!existsSync(join(outputsDir, artifactDir, file))) { + errors.push(`models/${f}: ${where} declared but artifact missing: outputs/${artifact}`) } }) } } -for (const entry of readdirSync(outputsDir, { withFileTypes: true })) { - if (entry.isDirectory() && !modelIds.has(entry.name)) { - errors.push(`outputs/${entry.name}/ exists but models/${entry.name}.json is missing`) - } -} - -// Every file under outputs// must be declared by a run — loose artifacts -// would otherwise ship without provenance. -for (const id of modelIds) { - const dir = join(outputsDir, id) - if (!existsSync(dir)) continue - const def = JSON.parse(readFileSync(join(modelsDir, `${id}.json`), 'utf8')) as { - runs?: Record> - } - const declared = new Set() - for (const [caseId, runRaw] of Object.entries(def.runs ?? {})) { - for (const run of Array.isArray(runRaw) ? runRaw : [runRaw]) { - declared.add(run?.file ?? `${caseId}.${caseKind.get(caseId)}`) - } - } - for (const file of readdirSync(dir)) { - if (!declared.has(file)) { - errors.push(`outputs/${id}/${file} is not declared by any run in models/${id}.json`) +// Every file under outputs/ must be declared by a run — loose artifacts would +// otherwise ship without provenance. +// `artifactDir` can be one or two path segments, so scan recursively and compare +// against the concrete paths declared above. +function walkOutputs(dir: string, prefix = '') { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.name.startsWith('.')) continue + const rel = prefix ? `${prefix}/${entry.name}` : entry.name + if (entry.isDirectory()) { + walkOutputs(join(dir, entry.name), rel) + } else if (!declaredArtifacts.has(rel)) { + errors.push(`outputs/${rel} is not declared by any run in models/*.json`) } } } +if (existsSync(outputsDir)) walkOutputs(outputsDir) if (errors.length) { console.error(`Data validation failed (${errors.length}):\n- ${errors.join('\n- ')}`)