Reorganize artifacts by model and harness

2026-06-13 12:59:01 +01:00
1 parent 4027ba94c2
commit a12017f31e
@@ -57,8 +57,8 @@ Pending: GLM-5.1 (PRs welcome)
 ## Structure

 ```
-outputs/<model-id>/<case-id>.<ext>   raw model artifacts (HTML / SVG), filename = case id
-models/<model-id>.json               combination registry: label / vendor / harness / effort / order / run notes
+outputs/<artifact-dir>/<case-id>.<ext>  raw model artifacts (HTML / SVG), filename = case id
+models/<agent-id>.json                  agent registry: label / vendor / harness / effort / artifactDir / order / run notes
 cases.json                           case definitions (bilingual prompts, maintainer-owned)
 scripts/validate-data.ts             data validation (CI runs it on every PR)
 ```
@@ -75,16 +75,16 @@ bun scripts/validate-data.ts   # the same data validation CI runs

 > **If you are an AI agent (Claude Code / Codex / Cursor, etc.), this part is for you:**
 >
-> - **The unit you contribute is one Agent = a model × the harness you run in.** The `<agent-id>` (lowercase letters / digits / dashes) encodes that pairing — e.g. `gpt-5-5-pro`, `claude-fable-5-cc` (cc = Claude Code). The same model in a different harness is a different Agent with a different id.
+> - **The unit you contribute is one Agent = a model × the harness you run in.** The `<agent-id>` (lowercase letters / digits / dashes) encodes that pairing — e.g. `gpt-5-5-pro`, `claude-fable-5-cc` (cc = Claude Code). The same model in a different harness is a different Agent with a different id. The `<agent-id>` is the vote and leaderboard identity; do not rename it just to reorganize files.
 > - **There are two cases right now**, defined in [`cases.json`](./cases.json): `mythos-craft` (HTML, a playable voxel world) and `pelican-cycling` (SVG, a pelican cycling by the sea). The prompt lives in `cases.json` — **use it verbatim, do not paraphrase**.
 > - **To find what's missing**: read `cases.json` (all cases) and `models/*.json` (what each Agent has run) — a `<case-id>` absent from an Agent's json is an open slot; any row in the Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill both.
 > - **Then follow the two-file flow below**, write a bilingual `note` (how the artifact was produced — one-shot or fixed), and run `bun scripts/validate-data.ts` until it passes before opening a PR.

 Contributions are data-only — no code changes needed:

-1. Artifact: `outputs/<model-id>/<case-id>.<ext>` (e.g. `outputs/gpt-5-5-pro/pelican-cycling.svg`).
-2. Registration: `models/<model-id>.json` (new file for a new combination; add to `runs` for an existing one) — see the schema example in the Chinese README.
+1. Artifact: preferably `outputs/<base-model>/<harness-effort>/<case-id>.<ext>` (e.g. `outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg`).
+2. Registration: `models/<agent-id>.json` (new file for a new combination; add to `runs` for an existing one) with `artifactDir` pointing at the artifact folder — see the schema example in the Chinese README.

-CI-enforced rules: dash-only lowercase ids matching the `outputs/` folder name; a **required bilingual provenance `note`** per run (which harness, what effort, one-shot or fixed); every declared run must have its artifact; multiple versions of the same combination x case go in an array with distinct `file` names (e.g. `pelican-cycling-2.svg`). Set `contributor` to your GitHub username — the site shows your avatar linked to your profile next to the run. Fill in `harness` and `effort` truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via [lobe-icons](https://github.com/lobehub/lobe-icons)) — contributors never touch icons.
+CI-enforced rules: dash-only lowercase `agent-id`s as stable identities; without `artifactDir`, artifacts default to `outputs/<agent-id>/`; with `artifactDir`, use one or two lowercase dash-case path segments, preferably `<base-model>/<harness-effort>`. A bilingual provenance `note` is required per run (which harness, what effort, one-shot or fixed); every declared run must have its artifact; multiple versions of the same combination x case go in an array with distinct `file` names (e.g. `pelican-cycling-2.svg`). Set `contributor` to your GitHub username — the site shows your avatar linked to your profile next to the run. Fill in `harness` and `effort` truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via [lobe-icons](https://github.com/lobehub/lobe-icons)) — contributors never touch icons.

 PRs get data validation from CI; merges to `main` rebuild the site automatically (usually instant, at most 6 hours).
@@ -57,8 +57,8 @@ NAGI STUDIO 的 LLM 测评案例集：同一段提示词，不同「模型 × Ha
 ## 结构 / Structure

 ```
-outputs/<model-id>/<case-id>.<ext>   模型产出原文件（HTML / SVG），文件名 = 案例 id
-models/<model-id>.json               组合登记：label / vendor / harness / effort / order / 各案例运行备注
+outputs/<artifact-dir>/<case-id>.<ext>  模型产出原文件（HTML / SVG），文件名 = 案例 id
+models/<agent-id>.json                  Agent 登记：label / vendor / harness / effort / artifactDir / order / 各案例运行备注
 cases.json                           案例定义：双语标题与提示词（维护者维护）
 scripts/validate-data.ts             数据校验（CI 对每个 PR 自动执行）
 ```
@@ -75,15 +75,15 @@ bun scripts/validate-data.ts   # 与 CI 相同的数据校验

 > **如果你是一个 AI Agent（Claude Code / Codex / Cursor 等），这一段是给你看的：**
 >
-> - **你贡献的单位是一个 Agent = 模型 × 你所在的 Harness。** `<agent-id>`（小写字母 / 数字 / 连字符）就编码这个组合，如 `gpt-5-5-pro`、`claude-fable-5-cc`（cc = Claude Code）。同一模型换个 Harness 就是另一个 Agent、另一个 id。
+> - **你贡献的单位是一个 Agent = 模型 × 你所在的 Harness。** `<agent-id>`（小写字母 / 数字 / 连字符）就编码这个组合，如 `gpt-5-5-pro`、`claude-fable-5-cc`（cc = Claude Code）。同一模型换个 Harness 就是另一个 Agent、另一个 id。`<agent-id>` 是投票与榜单身份，不要为了整理文件夹而改它。
 > - **当前有两个案例**，定义在 [`cases.json`](./cases.json)：`mythos-craft`（HTML，可玩体素世界）与 `pelican-cycling`（SVG，海边骑车的鹈鹕）。提示词以 `cases.json` 为准，**逐字使用、不要改写**。
 > - **想知道还缺哪些**：读 `cases.json`（全部案例）与 `models/*.json`（每个 Agent 跑过的案例）——某个 Agent 的 json 里缺某个 `<case-id>` 就是一个空位；上面 Registry 表里「产出」数小于案例总数的行，就是还缺案例的 Agent；全新 Agent 则两个案例都可补。
 > - **然后照下面两文件流程做**，写好双语 `note`（说明产出怎么来的、是否一次生成、是否有修复），提交前用 `bun scripts/validate-data.ts` 自检通过再发 PR。

 贡献**不需要改任何代码**，只涉及两类文件：

-1. 产出文件：`outputs/<model-id>/<case-id>.<ext>`（如 `outputs/gpt-5-5-pro/pelican-cycling.svg`）。
-2. 组合登记：`models/<model-id>.json`（新组合新建文件；已有组合在 `runs` 里加一条）：
+1. 产出文件：推荐放在 `outputs/<base-model>/<harness-effort>/<case-id>.<ext>`（如 `outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg`）。
+2. Agent 登记：`models/<agent-id>.json`（新组合新建文件；已有组合在 `runs` 里加一条），并用 `artifactDir` 指向产出目录：

 ```json
 {
@@ -91,6 +91,7 @@ bun scripts/validate-data.ts   # 与 CI 相同的数据校验
  "vendor": "OpenAI",
  "harness": "ChatGPT Web",
  "effort": "Extended Pro",
+  "artifactDir": "gpt-5-5-pro/chatgpt-web-extended-pro",
  "order": 20,
  "runs": {
    "pelican-cycling": {
@@ -105,7 +106,8 @@ bun scripts/validate-data.ts   # 与 CI 相同的数据校验
 ```

 规则（CI 自动校验，不满足会挂）：
- `model-id` 只用小写字母/数字/连字符（如 `doubao-seed-2-0-pro`），文件名与 `outputs/` 目录名一致；
+- `agent-id` 只用小写字母/数字/连字符（如 `doubao-seed-2-0-pro`），它是稳定身份；没有 `artifactDir` 时默认读取 `outputs/<agent-id>/`；
+- `artifactDir` 只用一层或两层小写 dash-case 路径（推荐 `<base-model>/<harness-effort>`），它只是产出文件位置，不影响投票与榜单身份；
 - 每条 run 的 `note` **双语必填**——这是本仓库的可信度来源，必须写明产出怎么来的；
 - 声明的 run 必须有对应的产出文件；
 - 同一组合对同一案例可提交多个版本：`runs.<case-id>` 写成数组，第二个版本起必须用 `file` 指定不同文件名（如 `pelican-cycling-2.svg`）；
@@ -3,6 +3,7 @@
  "vendor": "Anthropic",
  "harness": "Claude Code",
  "effort": "Max",
+  "artifactDir": "claude-fable-5/claude-code-max",
  "order": 12,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "Anthropic",
  "harness": "Cursor",
  "effort": "High",
+  "artifactDir": "claude-fable-5/cursor-high",
  "order": 13,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "Anthropic",
  "harness": "Claude Web App",
  "effort": "Max",
+  "artifactDir": "claude-fable-5/claude-web-app-max",
  "order": 10,
  "runs": {
    "mythos-craft": {
@@ -1,6 +1,9 @@
 {
  "label": "Claude Opus 4.8",
  "vendor": "Anthropic",
+  "harness": "Claude Code",
+  "effort": "Max",
+  "artifactDir": "claude-opus-4-8/claude-code-max",
  "order": 15,
  "runs": {
    "mythos-craft": [
@@ -37,7 +40,5 @@
        "contributor": "OHMFOHS"
      }
    ]
-  },
-  "harness": "Claude Code",
-  "effort": "Max"
+  }
 }
@@ -3,6 +3,7 @@
  "vendor": "Cursor",
  "harness": "Cursor",
  "effort": "Default",
+  "artifactDir": "composer-2-5/cursor-default",
  "order": 71,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "Cursor",
  "harness": "Grok Build TUI",
  "effort": "Default",
+  "artifactDir": "composer-2-5/grok-build-tui-default",
  "order": 72,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "Cursor",
  "harness": "Cursor",
  "effort": "Max",
+  "artifactDir": "composer-2-5/cursor-max",
  "order": 70,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "DeepSeek",
  "harness": "Claude Code",
  "effort": "Max",
+  "artifactDir": "deepseek-v4-pro/claude-code-max",
  "order": 90,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "ByteDance",
  "harness": "Doubao Web",
  "effort": "Fast Mode",
+  "artifactDir": "doubao-seed-2-0-mini/doubao-web-fast-mode",
  "order": 96,
  "runs": {
    "pelican-cycling": {
@@ -3,6 +3,7 @@
  "vendor": "ByteDance",
  "harness": "Doubao Web",
  "effort": "Pro Mode",
+  "artifactDir": "doubao-seed-2-0-pro/doubao-web-pro-mode",
  "order": 95,
  "runs": {
    "pelican-cycling": {
@@ -3,6 +3,7 @@
  "vendor": "Google",
  "harness": "Google AI Studio",
  "effort": "High",
+  "artifactDir": "gemini-3-1-pro/google-ai-studio-high",
  "order": 42,
  "runs": {
    "pelican-cycling": {
@@ -3,6 +3,7 @@
  "vendor": "Google",
  "harness": "AntiGravity",
  "effort": "High",
+  "artifactDir": "gemini-3-1-pro/antigravity-high",
  "order": 40,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "Google",
  "harness": "Google AI Studio",
  "effort": "High",
+  "artifactDir": "gemini-3-5-flash/google-ai-studio-high",
  "order": 52,
  "runs": {
    "pelican-cycling": {
@@ -3,6 +3,7 @@
  "vendor": "Google",
  "harness": "AntiGravity",
  "effort": "High",
+  "artifactDir": "gemini-3-5-flash/antigravity-high",
  "order": 50,
  "runs": {
    "mythos-craft": {
@@ -1,5 +1,6 @@
 {
  "label": "GLM-5.1",
  "vendor": "Zhipu AI",
+  "artifactDir": "glm-5-1/default",
  "order": 120
 }
@@ -3,6 +3,7 @@
  "vendor": "OpenAI",
  "harness": "ChatGPT Web",
  "effort": "Extended Pro",
+  "artifactDir": "gpt-5-5-pro/chatgpt-web-extended-pro",
  "order": 20,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "OpenAI",
  "harness": "Codex CLI",
  "effort": "xhigh",
+  "artifactDir": "gpt-5-5/codex-cli-xhigh",
  "order": 30,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "xAI",
  "harness": "Grok Build TUI",
  "effort": "Max",
+  "artifactDir": "grok-build/grok-build-tui-max",
  "order": 60,
  "runs": {
    "mythos-craft": {
@@ -1,9 +1,10 @@
 {
  "label": "Kimi K2.6",
  "vendor": "Moonshot AI",
-  "order": 100,
  "harness": "Kimi Code",
  "effort": "Thinking",
+  "artifactDir": "kimi-k2-6/kimi-code-thinking",
+  "order": 100,
  "runs": {
    "mythos-craft": {
      "note": {
@@ -1,9 +1,10 @@
 {
  "label": "Kimi K2.7-Code",
  "vendor": "Moonshot AI",
-  "order": 98,
  "harness": "Kimi Code",
  "effort": "Thinking",
+  "artifactDir": "kimi-k2-7-code/kimi-code-thinking",
+  "order": 98,
  "runs": {
    "pelican-cycling": {
      "note": {
@@ -1,9 +1,10 @@
 {
  "label": "MiniMax M3",
  "vendor": "MiniMax",
-  "order": 110,
  "harness": "MiniMax Code Web",
  "effort": "Thinking",
+  "artifactDir": "minimax-m3/minimax-code-web-thinking",
+  "order": 110,
  "runs": {
    "pelican-cycling": {
      "note": {
@@ -3,6 +3,7 @@
  "vendor": "Mistral AI",
  "harness": "Vibe",
  "effort": "Thinking",
+  "artifactDir": "mistral-medium-3-5/vibe-thinking",
  "order": 80,
  "runs": {
    "mythos-craft": {
@@ -3,6 +3,7 @@
  "vendor": "Nex-AGI",
  "harness": "Claude Code",
  "effort": "Max",
+  "artifactDir": "nex-n2-pro/claude-code-max",
  "order": 60,
  "runs": {
    "mythos-craft": {
@@ -1,6 +1,9 @@
 {
  "label": "Qwen3.7-Max",
  "vendor": "Alibaba",
+  "harness": "Qoder",
+  "effort": "Default",
+  "artifactDir": "qwen3-7-max/qoder-default",
  "order": 130,
  "runs": {
    "mythos-craft": {
@@ -17,7 +20,5 @@
      },
      "contributor": "Nagi-ovo"
    }
-  },
-  "harness": "Qoder",
-  "effort": "Default"
+  }
 }
@@ -3,6 +3,7 @@
  "vendor": "Xiaomi",
  "harness": "Claude Code",
  "effort": "Max",
+  "artifactDir": "mimo-v2-5-pro/claude-code-max",
  "order": 97,
  "runs": {
    "mythos-craft": {
@@ -16,6 +16,7 @@ const cases = JSON.parse(readFileSync(join(root, 'cases.json'), 'utf8')) as Arra
 const caseKind = new Map(cases.map((c) => [c.id, c.kind]))

 const ID_RE = /^[a-z0-9]+(-[a-z0-9]+)*$/
+const ARTIFACT_DIR_RE = /^[a-z0-9]+(-[a-z0-9]+)*(\/[a-z0-9]+(-[a-z0-9]+)*)?$/
 const errors: string[] = []

 const isBilingual = (v: unknown): boolean => {
@@ -25,15 +26,23 @@ const isBilingual = (v: unknown): boolean => {
 }

 const modelIds = new Set<string>()
+const artifactDirs = new Map<string, string>()
+const declaredArtifacts = new Set<string>()
 for (const f of readdirSync(modelsDir).filter((f) => f.endsWith('.json'))) {
  const id = f.replace(/\.json$/, '')
  modelIds.add(id)
  if (!ID_RE.test(id)) {
-    errors.push(`models/${f}: id "${id}" must be lowercase letters/digits/dashes only (it doubles as the outputs/ folder name)`)
+    errors.push(`models/${f}: id "${id}" must be lowercase letters/digits/dashes only (it is the stable voting key and default outputs folder)`)
  }

  type RunJson = { note?: unknown; file?: unknown; contributor?: unknown }
-  let def: { label?: unknown; vendor?: unknown; order?: unknown; runs?: Record<string, RunJson | RunJson[]> }
+  let def: {
+    label?: unknown
+    vendor?: unknown
+    artifactDir?: unknown
+    order?: unknown
+    runs?: Record<string, RunJson | RunJson[]>
+  }
  try {
    def = JSON.parse(readFileSync(join(modelsDir, f), 'utf8'))
  } catch (e) {
@@ -44,6 +53,19 @@ for (const f of readdirSync(modelsDir).filter((f) => f.endsWith('.json'))) {
  if (typeof def.label !== 'string' || !def.label.trim()) errors.push(`models/${f}: "label" is required`)
  if (typeof def.vendor !== 'string' || !def.vendor.trim()) errors.push(`models/${f}: "vendor" is required`)
  if (def.order !== undefined && typeof def.order !== 'number') errors.push(`models/${f}: "order" must be a number`)
+  if (def.artifactDir !== undefined && typeof def.artifactDir !== 'string') {
+    errors.push(`models/${f}: "artifactDir" must be a string`)
+  }
+  const artifactDir = typeof def.artifactDir === 'string' && def.artifactDir.trim() ? def.artifactDir.trim() : id
+  if (!ARTIFACT_DIR_RE.test(artifactDir)) {
+    errors.push(`models/${f}: artifactDir "${artifactDir}" must be one or two lowercase dash-case path segments, e.g. "claude-fable-5/claude-code-max"`)
+  }
+  const dirOwner = artifactDirs.get(artifactDir)
+  if (dirOwner) {
+    errors.push(`models/${f}: artifactDir "${artifactDir}" is already used by models/${dirOwner}.json`)
+  } else {
+    artifactDirs.set(artifactDir, id)
+  }

  for (const [caseId, runRaw] of Object.entries(def.runs ?? {})) {
    if (!caseKind.has(caseId)) {
@@ -61,6 +83,12 @@ for (const f of readdirSync(modelsDir).filter((f) => f.endsWith('.json'))) {
      if (run?.file !== undefined && typeof run.file !== 'string') {
        errors.push(`models/${f}: ${where}.file must be a string`)
      }
+      if (
+        typeof run?.file === 'string' &&
+        (run.file.includes('/') || run.file.includes('\\') || run.file.includes('..'))
+      ) {
+        errors.push(`models/${f}: ${where}.file must be a filename, not a path`)
+      }
      if (run?.contributor !== undefined && (typeof run.contributor !== 'string' || !run.contributor.trim())) {
        errors.push(`models/${f}: ${where}.contributor must be a non-empty GitHub username`)
      }
@@ -69,39 +97,31 @@ for (const f of readdirSync(modelsDir).filter((f) => f.endsWith('.json'))) {
        errors.push(`models/${f}: ${where} resolves to the same artifact "${file}" as another variant — extra variants must set a distinct "file"`)
      }
      seenFiles.add(file)
-      if (!existsSync(join(outputsDir, id, file))) {
-        errors.push(`models/${f}: ${where} declared but artifact missing: outputs/${id}/${file}`)
+      const artifact = `${artifactDir}/${file}`
+      declaredArtifacts.add(artifact)
+      if (!existsSync(join(outputsDir, artifactDir, file))) {
+        errors.push(`models/${f}: ${where} declared but artifact missing: outputs/${artifact}`)
      }
    })
  }
 }

-for (const entry of readdirSync(outputsDir, { withFileTypes: true })) {
-  if (entry.isDirectory() && !modelIds.has(entry.name)) {
-    errors.push(`outputs/${entry.name}/ exists but models/${entry.name}.json is missing`)
-  }
-}
-
-// Every file under outputs/<id>/ must be declared by a run — loose artifacts
-// would otherwise ship without provenance.
-for (const id of modelIds) {
-  const dir = join(outputsDir, id)
-  if (!existsSync(dir)) continue
-  const def = JSON.parse(readFileSync(join(modelsDir, `${id}.json`), 'utf8')) as {
-    runs?: Record<string, { file?: string } | Array<{ file?: string }>>
-  }
-  const declared = new Set<string>()
-  for (const [caseId, runRaw] of Object.entries(def.runs ?? {})) {
-    for (const run of Array.isArray(runRaw) ? runRaw : [runRaw]) {
-      declared.add(run?.file ?? `${caseId}.${caseKind.get(caseId)}`)
-    }
-  }
-  for (const file of readdirSync(dir)) {
-    if (!declared.has(file)) {
-      errors.push(`outputs/${id}/${file} is not declared by any run in models/${id}.json`)
+// Every file under outputs/ must be declared by a run — loose artifacts would
+// otherwise ship without provenance.
+// `artifactDir` can be one or two path segments, so scan recursively and compare
+// against the concrete paths declared above.
+function walkOutputs(dir: string, prefix = '') {
+  for (const entry of readdirSync(dir, { withFileTypes: true })) {
+    if (entry.name.startsWith('.')) continue
+    const rel = prefix ? `${prefix}/${entry.name}` : entry.name
+    if (entry.isDirectory()) {
+      walkOutputs(join(dir, entry.name), rel)
+    } else if (!declaredArtifacts.has(rel)) {
+      errors.push(`outputs/${rel} is not declared by any run in models/*.json`)
    }
  }
 }
+if (existsSync(outputsDir)) walkOutputs(outputsDir)

 if (errors.length) {
  console.error(`Data validation failed (${errors.length}):\n- ${errors.join('\n- ')}`)