Guide

AI Benchmarks

A plain-English guide to AI benchmarks: what an LLM benchmark is, how scores are produced, what the main benchmarks test, from the Artificial Analysis Intelligence Index and Arena to SWE-bench Pro and ARC-AGI, and why the same model gets different scores.

November 12, 2026 · The AI Rankings

Quick answer: An AI benchmark is a fixed set of tasks with known correct answers, used to score AI models on the same test so they can be compared. For an overall comparison, the Artificial Analysis Intelligence Index combines nine evaluations that Artificial Analysis runs on every model itself, with the same settings and with 60% of the weight on private test sets. For coding, SWE-bench Pro and Terminal-Bench give models real programming work and check it with tests. For which answers people prefer, the Arena leaderboard ranks models by blind public votes. The one caveat: a score depends on who ran the test and how. At the September 2026 launch of Claude Opus 5.5, Anthropic reported 66.4% on Terminal-Bench 4.0, while Artificial Analysis’s own run of the same model read 59.6%.

Definition: An AI benchmark is a standard test for AI models: a fixed set of questions or tasks, a scoring rule and a published result. Benchmarks for large language models, often called LLM benchmarks, cover knowledge, reasoning, maths, coding and agentic work. A score shows how a model did on that test, under those conditions, and nothing more.

This page explains the benchmarks themselves: what each one tests, how to read the numbers and how to judge which ones to trust. It does not rank models. For the current ranking and each model’s current scores, see best AI models; for coding tools, see best AI for coding; for how that ranking uses benchmarks, see how we rank. For how LLMs work in the first place, see what is an LLM, and for one-line definitions of the rest of the vocabulary, see the AI glossary.

What is an AI benchmark?

An AI benchmark is a test that every model takes under the same rules, so the results can be compared side by side.

Every benchmark has three parts:

• A task set. A fixed collection of problems: multiple-choice science questions, bugs in real code repositories, puzzles, or multi-step jobs in a computer terminal.
• A scoring rule. How an answer is marked. That can be an exact match against a known answer, a test suite that the model’s code must pass, or a grading rubric applied by a human or another AI model.
• A harness. The software that feeds the tasks to the model, gives it tools if the test allows them, and collects the answers. A different harness can move a score by several points, as the section on reading scores shows.

Benchmarks fall into three broad families:

An index, such as the Artificial Analysis Intelligence Index, is not a benchmark in itself. It is a weighted average of several benchmarks, designed to give one headline number.

AI benchmarks vs LLM benchmarks

The two terms mean almost the same thing. “LLM benchmarks” refers specifically to tests for large language models, which are the models behind ChatGPT, Claude and Gemini. “AI benchmarks” is the wider term and also covers image, video, speech and robotics tests. Most of the benchmarks people quote, and every benchmark on this page, are used to test LLMs.

The main AI benchmarks

These are the benchmarks most often quoted in model launches and on public leaderboards.

The Artificial Analysis Intelligence Index

The Artificial Analysis Intelligence Index is a single score that combines nine evaluations, run by Artificial Analysis on every model with the same prompts, settings and harness.

Artificial Analysis launched version 5 of the index on 27 October 2026. The biggest change is that 60% of the index now rests on evaluations with private test sets or held-out answers, which a model cannot have seen during training. Version 5 has four categories, each worth 25% of the score:

Source: Artificial Analysis methodology.

Reading the index

• Effort levels are listed separately. The index scores the same model at each reasoning-effort setting, and the setting changes both the score and the cost. When Artificial Analysis tested Claude Fable 5.2 at its 9 November 2026 launch, it scored 65 on version 5.0.1 at max effort, at $26.54 per index task, and 54 at low effort, at $3.10 per task (Artificial Analysis).
• Cost per task is part of the result. Artificial Analysis publishes what it cost to run each model through the index, counting reasoning tokens as well as answer tokens (Artificial Analysis).
• No single model leads every evaluation. When version 5 launched, Claude Opus 5.5 led three of the nine evaluations and five other models led the remaining six (Artificial Analysis).
• Versions are not comparable. Each version uses different tests, so a score on version 5 cannot be set against a score on version 4.3. When version 5 launched, Artificial Analysis put the best open-weight model 12 points behind the leader, against 3 points on version 4.3 (Artificial Analysis). For models you can download and run, see best open-source LLMs.

Arena: the human preference leaderboard

Arena, formerly called LMArena and before that Chatbot Arena, ranks AI models by blind human votes. A user types a prompt, two anonymous models answer, the user votes for the better answer, and only then are the models’ names revealed (Wikipedia).

• How the score works. Arena turns pairwise votes into a rating on an Elo-style scale, the system used to rate chess players. Its leaderboard moved from online Elo to a statistical method called the Bradley-Terry model in December 2023. A higher rating means a model wins more head-to-head votes.
• Where it came from. Chatbot Arena was released on 24 April 2023 by the research group LMSYS, moved to the lmarena.ai domain in September 2024, and rebranded as Arena on 28 January 2026. Its site is now arena.ai (Wikipedia).
• Who runs it now. LMArena incorporated as an independent company in April 2025 (Wikipedia).
• Beyond text. Arena added image comparisons in June 2024 and video in January 2026 (Wikipedia), so a model can rank differently on each kind of task.

What Arena measures: which answers people prefer, including tone, formatting and helpfulness, which automatic tests do not score.

What it does not measure: correctness. A vote records which answer a person preferred, so a confident, well-formatted wrong answer can win. In April 2025, Meta’s announcement of Llama 4 Maverick quoted an LMArena rating of 1417 for “an experimental chat version” of the model (Meta); that version differed from the one Meta released, and LMArena updated its leaderboard policies in response (Wikipedia). The same month, a study titled “The Leaderboard Illusion” found that the arena’s rules let some model providers test several private variants before a public release and retract scores if they chose.

Coding benchmarks: SWE-bench, SWE-bench Pro and Terminal-Bench

Coding benchmarks give a model a real programming task and check its work with tests, which makes them harder to game than multiple-choice questions.

SWE-bench and SWE-bench Verified

SWE-bench, first published in October 2023 by researchers at Princeton University and the University of Chicago, has 2,294 software engineering problems drawn from real GitHub issues across 12 popular Python repositories. The model must write a patch that resolves the issue, and the fix counts only if the repository’s tests pass once the patch is applied. SWE-bench Verified, a subset of 500 problems that software engineers confirmed are solvable, followed on 13 August 2024 as a collaboration with OpenAI.

In February 2026, OpenAI published a post titled “Why we no longer evaluate SWE-bench Verified”, saying the benchmark had lost its value as a measure of coding ability, for two reasons (The Decoder):

• Flawed tests. Many tasks reject correct solutions because their tests enforce specific implementation details or check functions the task does not describe.
• Contamination. GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash Preview could reproduce some of the original fixes from memory, a sign that the problems had leaked into training data.

OpenAI recommended SWE-bench Pro instead and said it was building its own non-public tests. Some labs kept publishing SWE-bench Verified scores after OpenAI’s post, so they still appear in launch announcements and on model pages; OpenAI’s findings apply to those scores too.

SWE-bench Pro

SWE-bench Pro, from Scale AI, has 1,865 problems drawn from 41 actively maintained repositories, in languages including Python, Go, JavaScript and TypeScript. It is split into a public set from 11 repositories, a held-out set from 12 and a commercial set from 18 proprietary codebases; the held-out and commercial problems are not published, which limits contamination. Scores on SWE-bench Pro come in two forms: the lab’s own harness, and Scale’s leaderboard, which runs models on its own agent scaffold. Check which one a quoted figure used.

Terminal-Bench

Terminal-Bench, hosted by Stanford University, Harbor and the Laude Institute, gives a model jobs to do in a computer terminal, across software engineering, system administration, data processing, model training and security, and checks the result with a test suite. Terminal-Bench 4.0, a set of 66 tasks, makes up 10% of version 5 of the Artificial Analysis Intelligence Index, launched in October 2026 (Artificial Analysis). Terminal-Bench shows how much the harness matters: at the 22 September 2026 launch of Claude Opus 5.5, Anthropic reported 66.4% on Terminal-Bench 4.0, while Artificial Analysis’s own run of the same model read 59.6%.

For which coding tools come out on top in practice, see best AI for coding and best AI code review tools.

Knowledge and reasoning benchmarks: MMLU, GPQA Diamond and Humanity’s Last Exam

Knowledge benchmarks are lists of hard questions with known answers. Each one was built to be harder than the last, because models kept reaching the top of the earlier ones.

MMLU

MMLU (Massive Multitask Language Understanding), published in September 2020 by researchers from UC Berkeley, Columbia University, the University of Chicago and the University of Illinois, has 15,908 multiple-choice questions across 57 subjects, from law to medicine to maths. It was the standard knowledge test for years. In June 2024, the authors of the harder MMLU-Pro wrote that model performance on benchmarks such as MMLU “has begun to plateau”, and a study the same month estimated that 6.49% of MMLU questions contain errors, including 57% of the virology questions it analysed. An MMLU score no longer separates frontier models.

GPQA Diamond

GPQA, published in November 2023 by researchers at New York University, Cohere and Anthropic, has 448 multiple-choice questions in biology, chemistry and physics, written by domain experts. Its Diamond subset is the 198 questions where both expert validators answered correctly and most non-experts answered wrongly. The paper calls the questions “Google-proof”: experts with or working towards PhDs scored 65%, while skilled non-experts scored 34% despite spending more than 30 minutes per question with unrestricted web access. Artificial Analysis removed GPQA Diamond from its index on 4 September 2026, describing it as saturated.

Humanity’s Last Exam

Humanity’s Last Exam (HLE), from the Center for AI Safety and Scale AI and first published in January 2025, has 2,500 questions from nearly 1,000 subject-expert contributors, across maths, the humanities and the natural sciences, with some questions including images. Artificial Analysis uses HLE-Diamond, a refined 1,000-question version split evenly between reasoning and knowledge questions. When Artificial Analysis launched version 5 of its index in October 2026, the top HLE-Diamond score was 68% (Artificial Analysis).

ARC-AGI: can a model learn something new?

ARC-AGI tests whether a model can work out an unfamiliar rule from a handful of examples, the kind of problem a person solves by noticing a pattern rather than by recalling a fact. François Chollet introduced the first version in 2019, and the ARC Prize Foundation runs three versions.

On ARC-AGI-3, 100% means an AI agent beats every game as efficiently as people do, measured against a controlled study of 458 human participants (ARC Prize).

ARC-AGI-3 also shows how much setup changes a score. The leaderboard also has a column for runs with a “provider adapter”, which keeps the model’s reasoning between requests and compacts long conversations. At GPT-6 Astra’s September 2026 launch, ARC Prize measured it at 62.7% on the standard harness at max effort and 99.9% with OpenAI’s provider adapter at high effort (ARC Prize). Claude Fable 5.2, tested on its 9 November 2026 launch day, scored 81.2% on the standard setting and 97.7% with the adapter (ARC Prize). The two columns put the models in opposite order, so always check which column a quoted ARC-AGI-3 figure comes from. For what “AGI” means in the first place, see what is AGI.

Agentic and long-task benchmarks: METR, FrontierMath and the rest

A newer group of benchmarks measures whether models can finish long, open-ended jobs rather than answer single questions.

• METR time horizon. METR, a non-profit that studies advanced AI agents, measures the length of task, in the time it takes human professionals, that a model can finish with 50% reliability. Its March 2025 study found that this time horizon had been doubling about every seven months for the previous six years; METR has since published updated estimates.
• FrontierMath. FrontierMath, from Epoch AI, is a set of several hundred unpublished, research-level maths problems written by mathematicians. Epoch AI has disclosed that OpenAI commissioned 300 of the problems, owns them and can see their solutions, apart from a holdout set for which OpenAI receives only the problem statements.
• Knowledge-work benchmarks. AA-Briefcase, AutomationBench-AA and GDP.pdf in the Artificial Analysis index test the jobs office workers do: multi-week projects, workflows across business software and answers grounded in long documents (Artificial Analysis).
• Hallucination benchmarks. AA-Omniscience scores factual accuracy and penalises incorrect guesses, and Vectara’s hallucination leaderboard measures how often a model introduces hallucinations when summarising a document. For why models make things up, see AI hallucinations.

How to read a benchmark score

The same model can get very different scores on the same benchmark. These are the five things to check before trusting a number.

A few more terms that come up in benchmark tables:

• pass@1 means the model gets one attempt per task, and the score is the share it gets right. pass@k gives it several attempts and counts a task as solved if any attempt succeeds, which produces a higher number.
• Private or held-out set means the questions are not published, so they cannot end up in training data. Artificial Analysis puts 60% of version 5 of its index on private sets or held-out answers for this reason.
• Cost per task is what it costs to run the model through a benchmark task, including hidden reasoning tokens. A model that is cheaper per token can cost more per task if it thinks for longer, so cost per task is the comparable figure between reasoning models. For per-token prices, see best LLM APIs.

Which AI benchmarks should you trust?

Independent, standardised runs are more comparable than vendor figures, because every model is tested the same way. Beyond that, match the benchmark to the job you care about.

Be more careful with:

• A lab’s own launch chart. The numbers are the lab’s own: it picks which benchmarks to show and runs its own harness. Read it as the vendor’s claim, and compare it with an independent run where one exists.
• Saturated benchmarks. Near-top scores on MMLU, GPQA Diamond and ARC-AGI-1 no longer tell frontier models apart.
• A single benchmark. Every benchmark measures a narrow slice. A model can lead one and trail on another, which is why indexes average several.
• Your own work. No benchmark is your workload. Giving two close models the same real tasks from your own work shows how each handles it.

Frequently asked questions

What are AI benchmarks?

AI benchmarks are standard tests for AI models: fixed sets of questions or tasks with a scoring rule, so different models can be compared on the same test. Common examples are the Artificial Analysis Intelligence Index, the Arena leaderboard, SWE-bench Pro for coding, Humanity’s Last Exam for expert knowledge and ARC-AGI for learning new rules.

What are LLM benchmarks?

LLM benchmarks are AI benchmarks for large language models, the models behind ChatGPT, Claude and Gemini. They test knowledge, reasoning, maths, coding and agentic tasks, and they are scored automatically against known answers, by test suites, or by human votes in the case of Arena.

What is the best AI benchmark?

No single benchmark covers everything. For an overall comparison, the Artificial Analysis Intelligence Index combines nine evaluations that Artificial Analysis runs on every model with the same settings, keeps 60% of the weight on private test sets or held-out answers, and publishes the cost per task. For coding, SWE-bench Pro and Terminal-Bench give models real programming work and check it with tests.

Which AI model has the highest benchmark score?

It depends on the benchmark: each one has its own leader, and the leaders change with almost every major release. The current ranking, with each model’s score on the Artificial Analysis Intelligence Index, is on best AI models.

What is the Artificial Analysis Intelligence Index?

The Artificial Analysis Intelligence Index is a single score that combines nine evaluations across knowledge work, coding, science and general knowledge, each category worth 25%. Version 5 launched on 27 October 2026 and puts 60% of its weight on evaluations with private test sets or held-out answers, which models cannot have seen in training.

What is LMArena?

LMArena, now called Arena, is a public leaderboard that ranks AI models by blind human votes: two anonymous models answer the same prompt, the user votes for the better answer, and the votes are turned into an Elo-style rating. It launched as Chatbot Arena in April 2023 and rebranded as Arena in January 2026.

What is SWE-bench?

SWE-bench is a coding benchmark that gives a model real GitHub issues from 12 Python repositories and counts a fix as correct only if the repository’s tests pass. Its 500-problem Verified subset was released with OpenAI in August 2024. In February 2026, OpenAI said it no longer evaluates SWE-bench Verified because of flawed tests and contamination, and recommended SWE-bench Pro instead.

What is Humanity’s Last Exam?

Humanity’s Last Exam is a benchmark of 2,500 questions from nearly 1,000 subject experts across maths, the humanities and the natural sciences, released by the Center for AI Safety and Scale AI in January 2025. Artificial Analysis uses a refined 1,000-question version, HLE-Diamond, in its index.

What is GPQA Diamond?

GPQA Diamond is the 198-question subset of GPQA, a set of graduate-level biology, chemistry and physics questions published in November 2023, made up of the questions both expert validators answered correctly and most non-experts got wrong. On the full set, PhD-level experts scored 65% and skilled non-experts with web access scored 34%. Artificial Analysis dropped it from its index in September 2026 as saturated.

What is ARC-AGI?

ARC-AGI is a benchmark that tests whether an AI can work out new rules from a few examples. ARC-AGI-1 and ARC-AGI-2 use grid puzzles, and top scores on both were above 95% on the ARC Prize leaderboard by November 2026. ARC-AGI-3, launched on 25 March 2026, uses 135 interactive games with no instructions, where 100% means beating every game as efficiently as people do.

Why do AI companies’ benchmark scores differ from independent ones?

AI companies run benchmarks on their own harness, with their own prompts, tools and effort settings, and choose which results to publish. Independent testers run every model the same way, which can produce a lower figure. At its September 2026 launch, Claude Opus 5.5 scored 66.4% on Terminal-Bench 4.0 on Anthropic’s harness and 59.6% in Artificial Analysis’s run.

Are AI benchmarks reliable?

AI benchmarks are reliable for comparing models on the specific task they test, under stated conditions, and unreliable as a general verdict. Their main weaknesses are contamination, where test questions leak into training data; saturation, where top models all score near 100%; and harness differences between vendor and independent runs.

What does benchmark saturation mean?

Benchmark saturation means the top models score so close to the ceiling that the benchmark can no longer tell them apart. MMLU, GPQA Diamond and ARC-AGI-1 are saturated. Artificial Analysis, for example, dropped GPQA Diamond from its index in September 2026 and now uses harder tests such as HLE-Diamond.

What is benchmark contamination?

Benchmark contamination happens when a benchmark’s questions or answers end up in a model’s training data, so the model can recall answers instead of solving the problems. OpenAI found that several frontier models could reproduce some of the original fixes for SWE-bench Verified tasks from memory, and private test sets are the main defence.

← All guides