Browse all visual benchmark suites

Compare artifacts within the same prompt and protocol. Counts describe published evidence, not quality scores.

Superbench 2.1

Superbench 2.0

1.x core

Specialist

Archived

Earlier published score comparison

Compare benchmark scores.

Choose two models and a workload to see where each leads.

September 2026 comparison

Provider-published results · checked

Read OpenAI’s source table

6 selected benchmarks · GPT-6 Astra vs Claude Fable 5.1 · pp = percentage points

Terminal-Bench 4.0

Coding · Higher is better
Astra57.9%
Gemini 3.8 Flash19.1%
38.8 ppAstra ahead

Terminal-Bench Science 0.1

Science · Higher is better
Astra64.6%
Gemini 3.8 FlashNot reported
No paired result

AutomationBench

Automation · Higher is better
Astra41.4%
Gemini 3.8 FlashNot reported
No paired result

Humanity’s Last Exam (with tools)

Reasoning · Higher is better
Astra57.2%
Gemini 3.8 FlashNot reported
No paired result

Artificial Analysis Intelligence Index v4.1.1

Reasoning · Higher is better · index points
Astra61.2
Gemini 3.8 Flash58.7
2.5 pointsAstra ahead

FrontierMath Tier 4 (v2)

Reasoning · Higher is better
Astra97.6%
Gemini 3.8 FlashNot reported
No paired result
All 40 benchmarks · 2 models View source snapshot
OpenAI launch table, checked 7 September 2026. — means not reported. * indicates a methodology note. Higher is better unless marked ↓.
BenchmarkAstraGemini 3.8 Flash
Terminal-Bench 4.057.9%19.1%
Terminal-Bench Science 0.1 *64.6%—
AutomationBench41.4%—
Humanity’s Last Exam (with tools)57.2%—
Artificial Analysis Intelligence Index v4.1.1 *61.258.7
FrontierMath Tier 4 (v2)97.6%—
DeepSWE v1.174.1%73.8%
FrontierCode 1.1 Extended (score) *64.5%56.3%
FrontierCode 1.1 Main (score) *53.3%43.6%
GPQA Diamond96.0%95.3%
ARC-AGI-295.0%—
ARC-AGI-198.5%—
Agents’ Last Exam59.3%—
OSWorld 2.0 · offline, v2026.08.08 *72.6%—
ScreenSpot-Pro (no tools) *92.7%—
BenchCAD (with tools) *95.9%—
BrowseComp91.5%—
OpenScore String Quartets (1 − OMR-NED)0.84—
Design tasks (OpenAI internal)50.0%—
Data science tasks (OpenAI internal)40.9%—
Database migration (OpenAI internal)63.9%—
Artificial Analysis Coding Agent Index v1.467.061.2
GeneBench Pro37.1%—
MedChemBench (OpenAI internal)49.3%—
LifeSciBench60.3%—
HealthBench Professional (length-adjusted) *63.4%52.1%
ExploitBench *100.0%—
ExploitGym *42.4%—
ExploitBench (June–August 2026) *39.0%—
SRE-Bench (single attempt) *88.0%—
SEC-Bench Pro85.4%—
Computer-use safety (OpenAI internal) ↓ *2.4%—
Computer-use safety with AutoReview ↓1.8%—
Circumvention (OpenAI internal) ↓0.0%—
ExploitGym honeypot ↓0.0%—
Impossible ExploitGym100.0%—
Hallucination (OpenAI internal) ↓4.2%—
MRCR v2 · 8 needles · 256K–512K100.0%—
MRCR v2 · 8 needles · 512K–1M96.3%—
ARC-AGI-3 *99.9%—
Sources and methodology

Maximum reported score at any effort; research/API environments. A shared table does not guarantee identical harnesses, budgets, or safeguards.

Gaps are differences in reported scores, not statistical significance. Percentage bars use 0–100; index bars use a 0–100 display scale.

Anthropic’s model report
Terminal-Bench Science 0.1
Anthropic reports standard error of about 3.5–4.5 points per model; small gaps may be noise.
Artificial Analysis Intelligence Index v4.1.1
Independent index as reproduced in OpenAI’s launch table; separate from the older v4.1 observations in the cost explorer.
FrontierCode 1.1 Extended (score)
Astra uses additional developer guidance about keeping code and tests scoped (source footnote 8).
FrontierCode 1.1 Main (score)
Astra uses additional developer guidance about keeping code and tests scoped (source footnote 8).
OSWorld 2.0 · offline, v2026.08.08
Partial score on the offline subset. Anthropic’s modified tasks and grading are incompatible; no Fable 5.1 value is supplied here.
ScreenSpot-Pro (no tools)
The source labels Mythos’s 87.3 as Fable 5 (footnote 17). It is excluded from the Fable column here because the model’s safeguards differ.
BenchCAD (with tools)
Claude results use three evaluation modifications. Scores shown for context; no direct gap is calculated.
HealthBench Professional (length-adjusted)
OpenAI re-evaluation with GPT-5.4 grading. Fable 5.1 uses Opus 5 fallback for refusals; not a standalone Fable result.
ExploitBench
Astra and Sol evaluated without production safeguards.
ExploitGym
The source’s Fable columns are Mythos 5.1 (30.4) and Mythos 5 (28.4), excluded here. Astra/Sol have no six-hour limit and no production safeguards. Not a production-model comparison.
ExploitBench (June–August 2026)
OpenAI internal set, without production safeguards. Sol’s 5.5 is affected by a 300-turn cap; source also reports 11.5 with fewer limits.
SRE-Bench (single attempt)
Cyber capability evaluation without production safeguards.
Computer-use safety (OpenAI internal)
Lower is better. Research harness without production confirmations; provider safeguards and tools still differ.
ARC-AGI-3
OpenAI Responses harness with two modified settings; see source footnote 1.
2models on shared source tables
3models on the same evaluation
3unranked for now

The September launch snapshot and retained independent measurements keep their own sources and versions; retired models are not presented as current.

Independent comparison

Earlier independent scores are now historical.

No models in the current roster have a compatible retained Artificial Analysis Index v4.1 row. The historical measurements remain attached to their archived model records; no score is inferred for Kimi K3.1 Pro.

Cost explorer

Estimate your token costs.

Set total input and output tokens for a workload. This estimate uses the displayed short-context API rates, including active promotions, across all requests.

Example workloads

Opus 5.5 is $4 input / $20 output; Astra and Fable 5.1 are $10 / $50; Sol is $2 / $10. Astra and Sol requests above 272K input tokens use their higher rates for the full request. Grok 4.7 requests at or above 200K prompt tokens use $4 / $12. DeepSeek uses peak rates here; off-peak rates are half. Cache reads, cache writes, tools, regional premiums, and Fast mode are excluded.

ModelAA IndexInput / 1MOutput / 1MSpeedYour task
Claude Opus 5.5Anthropic · Standard API · pricing checked 2026-09-23—See v4.1.1 above$4.00$20.00Not measuredNo speed estimate$2.40Price ↗
GPT-6 AstraOpenAI · Standard API · pricing checked 2026-09-08—See v4.1.1 above$10.00$50.00Not measuredNo speed estimate$6.00Price ↗
GPT-6 SolOpenAI · Standard API · pricing checked 2026-09-23—See v4.1.1 above$2.00$10.00Not measuredNo speed estimate$1.20Price ↗
DeepSeek V4.1 Flash-1017DeepSeek · Standard API · pricing checked 2026-10-25—See v4.1.1 above$0.30$1.20Not measuredNo speed estimate$0.16Price ↗
Claude Fable 5.1Anthropic · Standard API · pricing checked 2026-09-08—See v4.1.1 above$10.00$50.00Not measuredNo speed estimate$6.00Price ↗
Grok 4.7SpaceXAI · Standard API · pricing checked 2026-09-22—See v4.1.1 above$2.00$6.00Not measuredNo speed estimate$0.88Price ↗

What is included: uncached input and generated output at short-context API rates. The baseline full-catalog audit was 2026-09-08; newer additions retain their own verification date in the catalog.

What is not included: long-context premiums, cache writes, tool calls, storage, batch discounts, and provider markups. A long prompt may cost more than this estimate; check the linked pricing source.

Blended reference: The 7:2:1 cache-hit/input/output mix has no current retained-AA rows to summarize. Scenario costs above still use each current catalog’s published rates.

Comparable score lanes

Keep incompatible scores separate.

Each plot covers one benchmark. Orange marks one source table. Outlined marks use the same named evaluation from another publisher and are not assumed to be identical runs.

Agentic coding

SWE-bench Pro

GPT-5.6 comparison table ↗

Public tasks. Scores below are published in cross-model tables; effort and harness are retained in each source.

  1. Inkling
    54.3%Same evaluation ↗

Published July 2026. Open the source before using these figures for a purchasing decision.

Terminal agents

Terminal-Bench 2.1

GPT-5.6 comparison table ↗

Command-line task completion. The primary figures use OpenAI’s shared table; the other reported scores are labelled separately rather than normalized.

  1. Inkling
    63.8%Same evaluation ↗

Published July 2026. Open the source before using these figures for a purchasing decision.

Science reasoning

GPQA Diamond

Inkling model card cross-model table ↗

No-tools science questions. This lane keeps the reported configuration in view because reasoning budgets differ by provider.

  1. Inkling
    87.2%Shared table

Published July 2026. Open the source before using these figures for a purchasing decision.

Methodology

Limits of this comparison

Artificial Analysis Index results come from one independent harness. The exact model configuration remains attached to every score.

Token cost scenarios multiply published input and output prices by the workload you enter. They exclude tools, storage, cache writes, and provider discounts.

A shared-table mark means the source published the compared models in one table. A same-eval mark means the benchmark name matches, but the source or configuration differs.

Missing data is not a low score. Models without a compatible public score remain unranked until a reproducible result is published.

Our Superbash visual runs are a separate, same-prompt evidence layer. Use the live run comparison to inspect build quality directly.

Inspect the same-prompt visual runs →

Edition coverage

Recorded means a published artifact, not a behavioral pass. Historical returning results retain their original protocol.

deepseek-v4-1-flash-1017 · Superbench 2.0: 4/4 recorded · 0 pending · 0 not run
  • 17 Prism Freight Pavilion · single-prompt · Recorded
  • 18 Tidal Ledger Observatory · single-prompt · Recorded
  • 19 Last-Light Relay · single-prompt · Recorded
  • 20 Quarantine Signal Desk · single-prompt · Recorded
claude-fable-5-2 · Superbench 2.1: 6/6 recorded · 0 pending · 0 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Recorded
  • 22 Concourse Divide · single-prompt · Recorded
  • 23 Lightkeeper's Atelier · single-prompt · Recorded
  • 24 Wrecklight Salvage · single-prompt · Recorded
deepseek-v4-1-flash · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
deepseek-v4-pro · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
fable-5 · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
fable-5-1 · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
gemini-3-7-flash · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
glm-5-3 · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
glm-5-3-flash · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
gpt-6-1-terra · Superbench 2.1: 6/6 recorded · 0 pending · 0 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Recorded
  • 22 Concourse Divide · single-prompt · Recorded
  • 23 Lightkeeper's Atelier · single-prompt · Recorded
  • 24 Wrecklight Salvage · single-prompt · Recorded
gpt-6-2-astra · Superbench 2.1: 6/6 recorded · 0 pending · 0 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Recorded
  • 22 Concourse Divide · single-prompt · Recorded
  • 23 Lightkeeper's Atelier · single-prompt · Recorded
  • 24 Wrecklight Salvage · single-prompt · Recorded
gpt-6-2-sol · Superbench 2.1: 6/6 recorded · 0 pending · 0 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Recorded
  • 22 Concourse Divide · single-prompt · Recorded
  • 23 Lightkeeper's Atelier · single-prompt · Recorded
  • 24 Wrecklight Salvage · single-prompt · Recorded
gpt-6-astra · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
gpt-6-luna · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
gpt-6-sol · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
grok-4-6 · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
grok-4-7 · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
kimi-code · Superbench 2.1: 6/6 recorded · 0 pending · 0 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Recorded
  • 22 Concourse Divide · single-prompt · Recorded
  • 23 Lightkeeper's Atelier · single-prompt · Recorded
  • 24 Wrecklight Salvage · single-prompt · Recorded
minimax-m3-1-flash · Superbench 2.1: 6/6 recorded · 0 pending · 0 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Recorded
  • 22 Concourse Divide · single-prompt · Recorded
  • 23 Lightkeeper's Atelier · single-prompt · Recorded
  • 24 Wrecklight Salvage · single-prompt · Recorded
opus-5-5 · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
qwen-3-8-max · Superbench 2.1: 2/6 recorded · 0 pending · 4 not run
  • 15 Ember Glider · repository-run · Recorded
  • 14 Low-Poly Tower Defense · repository-run · Recorded
  • 21 Skyhook Cargo · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 22 Concourse Divide · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 23 Lightkeeper's Atelier · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.
  • 24 Wrecklight Salvage · single-prompt · Not run — Historical returning-brief artifacts only; the four new 2.1 contracts have not been published for this model version.