Add AGENTS.md + tighten contribution flow
- AGENTS.md: public contributor/agent guide (two-file rule, model-centric ids, one-entry-per-harness, bilingual notes, screenshot proof) - PR template: require a generation proof screenshot; drop the manual update-registry step; add id-naming and no-README-edit checks - CI: PRs only run validate-data; pushes to main auto-regenerate and commit the README registry (contents:write on that job, [skip ci], no loop) - README/update-registry: reflect auto-synced registry; fix stale 'two cases' -> three (mythos-craft, pelican-cycling, skeleton-watch); link AGENTS.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
2fb8499fa8
commit
30983f0e7d
6 files changed
+167
-26
No files matched your search
+8
-6
@@ -25,7 +25,7 @@ One-shot LLM eval cases by NAGI STUDIO: same prompt, different model x harness x
|
||||
## Registry
|
||||
|
||||
<!-- registry:start -->
|
||||
> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI checks that it stays current.
|
||||
> This table is generated from `models/*.json` by `bun scripts/update-registry.ts`; CI re-syncs it automatically after merge — contributors need not hand-edit it.
|
||||
|
||||
| Model | Vendor | Harness x Effort | Runs |
|
||||
|---|---|---|---|
|
||||
@@ -77,7 +77,7 @@ outputs/<artifact-dir>/<case-id>.<ext> raw model artifacts (HTML / SVG), filena
|
||||
models/<agent-id>.json agent registry: label / vendor / harness / effort / artifactDir / order / run notes
|
||||
cases.json case definitions (bilingual prompts, maintainer-owned)
|
||||
scripts/validate-data.ts data validation (CI runs it on every PR)
|
||||
scripts/update-registry.ts README Registry table generation (CI checks freshness)
|
||||
scripts/update-registry.ts README Registry table generation (CI auto-syncs after merge)
|
||||
```
|
||||
|
||||
> The site and vote-backend source lives in the private `nagi-bench-site` repo, deployed on Cloudflare Pages; this repo is the site's data source.
|
||||
@@ -85,17 +85,19 @@ scripts/update-registry.ts README Registry table generation (CI checks
|
||||
## Validation
|
||||
|
||||
```bash
|
||||
bun scripts/update-registry.ts # updates README Registry tables from models/*.json
|
||||
bun scripts/validate-data.ts # the same data validation CI runs
|
||||
bun scripts/validate-data.ts # the same data validation CI runs (run this before submitting)
|
||||
bun scripts/update-registry.ts # optional: preview the Registry tables locally; CI syncs them after merge
|
||||
```
|
||||
|
||||
## Contributing a run
|
||||
|
||||
> Full rules live in [**AGENTS.md**](./AGENTS.md) (the contributor / AI-agent guide).
|
||||
|
||||
> **If you are an AI agent (Claude Code / Codex / Cursor, etc.), this part is for you:**
|
||||
>
|
||||
> - **The unit you contribute is one Agent = a model × the harness you run in.** The `<agent-id>` (lowercase letters / digits / dashes) encodes that pairing — e.g. `gpt-5-5-pro`, `claude-fable-5-cc` (cc = Claude Code). The same model in a different harness is a different Agent with a different id. The `<agent-id>` is the vote and leaderboard identity; do not rename it just to reorganize files.
|
||||
> - **There are two cases right now**, defined in [`cases.json`](./cases.json): `mythos-craft` (HTML, a playable voxel world) and `pelican-cycling` (SVG, a pelican cycling by the sea). The prompt lives in `cases.json` — **use it verbatim, do not paraphrase**.
|
||||
> - **To find what's missing**: read `cases.json` (all cases) and `models/*.json` (what each Agent has run) — a `<case-id>` absent from an Agent's json is an open slot; any row in the script-generated, CI-checked Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill both.
|
||||
> - **There are three cases right now**, defined in [`cases.json`](./cases.json): `mythos-craft` (HTML, a playable voxel world), `pelican-cycling` (SVG, a pelican cycling by the sea), and `skeleton-watch` (HTML, a skeleton mechanical watch). The prompt lives in `cases.json` — **use it verbatim, do not paraphrase**.
|
||||
> - **To find what's missing**: read `cases.json` (all cases) and `models/*.json` (what each Agent has run) — a `<case-id>` absent from an Agent's json is an open slot; any row in the script-generated Registry table above whose "Runs" count is below the number of cases is an Agent missing a case; a brand-new Agent can fill them.
|
||||
> - **Then follow the two-file flow below**, write a bilingual `note` (how the artifact was produced — one-shot or fixed), and run `bun scripts/validate-data.ts` until it passes before opening a PR.
|
||||
|
||||
Contributions are data-only — no code changes needed:
|
||||
|
||||
Reference in new issue
Block a user