Research protocol

Methods & limitations

ACBench measures a system under one frozen Agent Command protocol. It does not measure general intelligence. The primary official comparison holds one PlayerThree harness/version, prompts, tools, practice permissions and budget fixed across models. Harness identity is recorded but is not a primary leaderboard dimension. Native or alternate harnesses require distinct system-comparison suites.

Opponents and scoring

Each authoring replicate starts independently. Frozen policies face versioned built-in opponents on predeclared maps and mirrored world blocks. Score is 100 times the fixed weighted mean of win + 0.5 draw. Runs receive equal weight; model-caused authoring failure scores zero. Infrastructure gaps (outages, setup failures, interruptions) keep results incomplete rather than scoring zero or shrinking the denominator. Timeouts follow the frozen adjudication.

Uncertainty

Crossed run/block bootstrap resamples independent authoring runs and shared world blocks while preserving strata and mirrors. Manifests freeze algorithm, seed and replicate count. A single authoring run cannot estimate authoring variance; absent intervals remain unavailable. Per-run scores, attempts and independent blocks are separate counts. Ordering does not establish significant differences.

Charts

Each version leads with its ranked scores and 95% intervals. Win rate by opponent scores each opponent like the headline (frozen map weights, runs equally weighted) and is drawn only when every evaluation permutation names its opponent; a missing point means unscored games. Run-to-run variance shows every scored authoring run, its min–max, mean and ±1 sample standard deviation. Nothing missing is drawn as zero.

Cost

Every run reports its authoring cost with scope, source, currency, pricing date and version; the results table shows the mean per system. Unknown, partial or unpriced cost reads as unavailable, never zero. Runner estimates are not invoices; evaluation compute is excluded. Failed runs remain in the denominator. Catalogue prices never rewrite historical bills.

Tracks

Official authored-policy is curated runner-observed authoring. Community evaluates immutable submitted artifacts: a trusted rerun establishes how the artifact plays, not which model wrote it. Live-command is a separate realtime task; no live-command protocol or decision-barrier execution is advertised here. Requested provider/model metadata is not verified model weights.

Versions

Each ACBench version (v0.1, v0.2, …) freezes one suite: opponents, maps, budgets and scoring. A new version gets a new permanent page; earlier versions stay published, so scores compare only within a version. A correction appends a revision of the same version with its reason.

Reproducibility & privacy

Versioned downloads contain suite, system, policy and publication digests, scoring rows, failure lineage and immutable corrections. Active grading seeds remain protected; block IDs preserve auditability. Publication is an explicit archival opt-in independent of expiring private Series results. No bearer shares, owner messages, secrets, private prompts or hidden chain-of-thought belong in public evidence. Raw timelines are not playable replays.