ACBenchPOLICY BENCHMARK

community · runner-observed · revision 1

GPT-6.1 Sol

via OpenRouter

ReleaseVersioned evidence JSONManifest
Fixed-reference score
13.3
95% CI 0.0–26.7 · complete
Independent runs
2 / 2
0 failed authoring runs retained
Scored games
60 / 60
3 independent world blocks
Mean authoring cost
$2.39
Exact 2.39360125 USD · evaluation compute excluded

Independent run distribution

Score 0–100 · one bar per declared run, not a confidence interval. Failed authoring remains in the score.

  1. run-10.0
    evaluated · cutoff
  2. run-226.7
    evaluated · cutoff

Recorded game outcomes

60 recorded trial rows · 60 expected games. Raw counts, not weighted scores.

win
8
draw
0
loss
52
forfeit
0
timeout
0
incomplete
0

Timeouts follow frozen adjudication. Missing games are not draws; incomplete cells are not renormalized.

Run evidence and methods

Fixed-reference scores · 0–100 · descriptive ordering, not a significance test
Independent runsScored / expected gamesEvidence
GPT-6.1 Solopenai/gpt-6.1-sol · via OpenRouter13.30.0–26.7
Complete
2/20 failed · 3 blocks60/60$2.39Evaluation compute excludedcommunity · runner-observedRun evidence

Artifact performance is not model-authorship verification. Requested provider/model metadata does not verify model weights.

Authoring lineage, failures & cost coverage
Every declared run, including failures; score values are not confidence intervals.
RunStatus / failureScorePolicy SHA-256AttemptsCost / coverage / pricing
run-1evaluatedcutoff0.0eb491d0426acb93570b438e235e2e6efbcfea21d2a1fed4a6bc6a99d489e43baSeries 6b079df6-d750-4e58-9a44-b53e5e60a800 · revision 62302 authoring$2.74authoring-run · complete · router-reportedopenrouter-catalog/2026-11-13T03:30:00Z+pinned-hosts · 2026-11-13T03:30:00.000Z
run-2evaluatedcutoff26.787f05cc005157e5400c5d0d4fe83eac4bf63bee07a54288f31c4dcc7d9127fccSeries 243b64a8-7022-456a-93fb-4befce73affe · revision 62301 authoring$2.05authoring-run · complete · router-reportedopenrouter-catalog/2026-11-13T03:30:00Z+pinned-hosts · 2026-11-13T03:30:00.000Z
Opponent / map strata
Frozen weights; incomplete cells are not renormalized
StratumWeightWin / draw / loss / forfeit / timeout / incomplete
crossfire14 / 0 / 16 / 0 / 0 / 0
twin_fords12 / 0 / 18 / 0 / 0 / 0
crown12 / 0 / 18 / 0 / 0 / 0
Frozen methods, hashes & attribution
Suite
acbench-v1-ed46598e332a 1
Game patch
0.1.0
Controlled roles
depths, surface, sky, orbit
Model (requested / declared)
openai/gpt-6.1-sol
Provider
openai
Fixed harness
opencode/1.19.5
Provenance
runner-observed
Publication SHA-256
5bdbe734b78d2d4b3df69ef0681634e62c0948ed2868692fcc0d35e507261248
Suite SHA-256
9e7ab25c84cac9e28bd1d5b4cb1809ffe216ef656cf6e109277bf00d8bcf589c
Configuration SHA-256
d748a6577e82b8f4de98dd470a5cf80eb4bad39f0e49f06c14286b3f2f62139c
Released
2026-11-14T23:38:19.698Z
License
CC-BY-4.0

Aggregate method: crossed-run-block-bootstrap/1, 2000 bootstrap replicates, seed 1729.

{
  "id": "acbench-v1-ed46598e332a",
  "game": "agent-command",
  "build": {
    "gameDigest": "7374d9bdb99cb98f2979cc79217f019c04fe806bc92f7a9da2ecfbc15f05a4ff",
    "runnerDigest": "eb01d29a12a07facf6b5868607f5e80f1d7fabd3595efac9d42301668d7042e3"
  },
  "roles": [
    "depths",
    "surface",
    "sky",
    "orbit"
  ],
  "title": "ACBench re-score: four-plane authored policies, 45 min wall clock, opencode 1.19.5; novice/standard/veteran/elite/champion bots both sides on crossfire, twin_fords, crown × 1 seed(s)",
  "track": "authored-policy",
  "blocks": [
    {
      "blockId": "crossfire-seed-5101",
      "stratumId": "crossfire",
      "scenarioId": "crossfire-seed-5101"
    },
    {
      "blockId": "twin_fords-seed-5101",
      "stratumId": "twin_fords",
      "scenarioId": "twin_fords-seed-5101"
    },
    {
      "blockId": "crown-seed-5101",
      "stratumId": "crown",
      "scenarioId": "crown-seed-5101"
    }
  ],
  "digest": "9e7ab25c84cac9e28bd1d5b4cb1809ffe216ef656cf6e109277bf00d8bcf589c",
  "strata": [
    {
      "id": "crossfire",
      "weight": 1
    },
    {
      "id": "twin_fords",
      "weight": 1
    },
    {
      "id": "crown",
      "weight": 1
    }
  ],
  "harness": {
    "id": "opencode",
    "version": "1.19.5"
  },
  "scoring": {
    "method": "weighted-score/1",
    "timeout": "draw",
    "authoringFailure": "zero"
  },
  "version": 1,
  "authoring": {
    "budgets": {
      "maxTurns": 20000,
      "maxTokens": 200000000,
      "maxCostUsd": "10"
    },
    "maxWallMs": 2700000,
    "replicates": 2,
    "practiceSeeds": [
      17,
      29
    ]
  },
  "evaluation": {
    "repetitions": 1,
    "permutations": [
      "novice-nw",
      "novice-se",
      "standard-nw",
      "standard-se",
      "veteran-nw",
      "veteran-se",
      "elite-nw",
      "elite-se",
      "champion-nw",
      "champion-se"
    ]
  },
  "references": [
    {
      "id": "novice",
      "digest": "da555df45a2fee0457c16c2846c788bfc58fd69417103f171b5570be1b9bb8fd"
    },
    {
      "id": "standard",
      "digest": "043f62745696ead3341490c9a74d02747d1764ac2d448d37cdab9a9e75deac36"
    },
    {
      "id": "veteran",
      "digest": "5ee97857e5a647d38c10ee781f3d16b707cc7201df185a8777e589a6cee7af1f"
    },
    {
      "id": "elite",
      "digest": "288b785872251ea06d68d018d3fb58c75d6850ca7b13de3f7ef9ae7b28c5e2b6"
    },
    {
      "id": "champion",
      "digest": "c7ab19194d52945b388b5ba554c605cf0dba49c44eec12b0fbc33a2d66cd7ac1"
    }
  ],
  "gameVersion": "0.1.0",
  "surfaceHash": "c62c6556b9f21ab26e1be42dd0161dc58f2048d86cb2efc6a62512522560dca8",
  "uncertainty": {
    "seed": 1729,
    "method": "crossed-run-block-bootstrap/1",
    "replicates": 2000
  },
  "seriesDigest": "e2d9fc69cd48f557fa195e997c29b3acf7e23981c3d01280db3874d207138a9a"
}
Cite this revision

ACBench. gpt-6.1-sol (via OpenRouter). acbench-v1-ed46598e332a 1. Publication publication-e78b58e269267e44cd1c8f935c0a0dc2, revision 1. SHA-256 5bdbe734b78d2d4b3df69ef0681634e62c0948ed2868692fcc0d35e507261248.

Retention: until withdrawn. Downloaded copies cannot be recalled. Raw timeline, source and private transcripts are not included in this public projection.