A categorized field guide of LLM benchmark families shown as labeled lanes on a clean data dashboard, including a lane for speed and throughput

Blog · By Anton Gulin · Updated June 28, 2026

Examples of LLM Benchmarks: A Field Guide (Quality + Speed)

Last updated: 2026-06-28

Ask “what are examples of LLM benchmarks?” and most lists hand you a wall of acronyms with no structure. This field guide fixes that. It groups real, currently used benchmarks into seven categories, reasoning, coding, math, knowledge and truthfulness, agentic, human-preference, and speed, and gives each entry what it measures, who maintains it, and one source you can check. The last category, speed, is the one nearly every list omits.

What are examples of LLM benchmarks?

Examples of LLM benchmarks include MMLU-Pro and GPQA Diamond for reasoning, SWE-bench Verified and LiveCodeBench for coding, AIME and MATH for mathematics, SimpleQA and TruthfulQA for factuality, tau-bench and WebArena for agents, LMArena for human preference, and tokens-per-second plus time-to-first-token for speed. Each measures a distinct, non-interchangeable skill.

The first lesson of any benchmark list is that the categories matter more than the names. A model can top a reasoning leaderboard and fail a coding test, or generate brilliant answers far too slowly to ship. The table below summarizes every benchmark in this guide; the sections that follow explain each category, its grading method, and what a strong score actually tells you. For the underlying concept, see What Is an LLM Benchmark?.

BenchmarkCategoryWhat it measuresMaintainerSource
MMLU-ProReasoningBroad knowledge, 10-option MCQTIGER-Lab (Wang et al., 2024)arXiv:2406.01574
GPQA DiamondReasoningPhD-level “Google-proof” scienceRein et al. (NYU), 2023arXiv:2311.12022
MMLUReasoning57-subject knowledge baselineHendrycks et al., 2020arXiv:2009.03300
SWE-bench VerifiedCodingResolving real GitHub issuesOpenAI / Jimenez et al., 2024openai.com
HumanEvalCodingPython function correctnessChen et al. (OpenAI), 2021arXiv:2107.03374
LiveCodeBenchCodingContamination-free contest codeJain et al., 2024arXiv:2403.07974
AIME 2025MathOlympiad-level competition mathArtificial Analysis (tracker)artificialanalysis.ai
MATHMath12,500 competition problemsHendrycks et al., 2021arXiv:2103.03874
SimpleQAKnowledgeShort-form factual accuracyOpenAI (Wei et al., 2024)openai.com
TruthfulQAKnowledgeResistance to misconceptionsLin et al., 2022arXiv:2109.07958
tau-benchAgenticTool-agent-user interactionSierra Research, 2024arXiv:2406.12045
WebArenaAgenticAutonomous web tasksZhou et al. (CMU), 2023arXiv:2307.13854
LMArenaHuman preferenceBlind pairwise voting (Elo-like)Arena / UC Berkeley, 2023lmarena.ai
Tokens/sec + TTFTSpeedThroughput and latencyArtificial Analysisartificialanalysis.ai

Which benchmarks measure reasoning?

Reasoning benchmarks test whether a model recalls facts and works through multi-step problems across many subjects. They are usually multiple-choice and graded against an answer key, which makes them cheap to run at scale. The leading current examples are MMLU-Pro and GPQA Diamond, both built specifically because the original MMLU saturated near 92 to 94 percent for frontier models (Source: TIGER-Lab, 2024).

MMLU-Pro and the original MMLU

MMLU (Massive Multitask Language Understanding) was released by Hendrycks et al. in 2020 and became the default knowledge test, covering 57 subjects from elementary math to professional law (Source: arXiv, 2020). When frontier models saturated it, TIGER-Lab introduced MMLU-Pro in 2024, expanding to over 12,000 reasoning-heavy questions and raising the answer choices from four to ten, which cut top-model accuracy by roughly 16 to 33 points and spread the field back out (Source: TIGER-Lab, arXiv:2406.01574, 2024).

GPQA Diamond

GPQA Diamond is the highest-quality 198-question subset of the Graduate-Level Google-Proof Q&A benchmark, authored by Rein et al. at NYU in 2023 (Source: Rein et al., arXiv:2311.12022, 2023). Its questions cover graduate physics, biology, and chemistry, written so that non-expert PhD holders score around 34 percent even with web access. That “Google-proof” design makes it one of the few reasoning tests still spreading frontier models across a wide range rather than clustering them near the ceiling.

Which benchmarks measure coding?

Coding benchmarks measure whether a model produces correct, runnable code, graded by executing unit tests rather than matching text. This is stricter than multiple choice: code either passes or fails. The category spans short function-writing tasks (HumanEval), contamination-resistant contest problems (LiveCodeBench), and full GitHub-issue resolution in real repositories (SWE-bench Verified). For a deeper look, see the LLM Coding Benchmark guide.

SWE-bench Verified

SWE-bench Verified is the gold standard for evaluating coding agents. It uses 500 real GitHub issues from popular Python repositories, human-verified so each issue is clear and each fix is testable. The original SWE-bench came from Jimenez et al. in 2023; OpenAI curated the human-validated Verified subset in 2024 to make scoring more reliable (Source: OpenAI, 2024). A model must read the issue, navigate the codebase, and produce a working patch, so reported scores depend heavily on the agent scaffolding around the model.

HumanEval and LiveCodeBench

HumanEval, introduced by Chen et al. at OpenAI in 2021, contains 164 hand-written Python problems graded by executing test cases, the first coding benchmark to verify by execution rather than text similarity (Source: Chen et al., arXiv:2107.03374, 2021). It is now saturated above 90 percent. LiveCodeBench, from Jain et al. in 2024, fixed that by scraping contest problems from LeetCode, Codeforces, and AtCoder only after each model’s training cutoff, so models cannot have memorized the answers (Source: Jain et al., arXiv:2403.07974, 2024).

Which benchmarks measure mathematics?

Math benchmarks test multi-step quantitative reasoning with verifiable numeric answers, which makes grading objective: an answer is right or wrong. The two anchors are the MATH dataset for breadth and AIME for olympiad-level difficulty. Both have largely saturated at the very top, so a one- or two-point gap among the leading models is closer to noise than a real capability difference (Source: BenchLM, 2026).

MATH and AIME 2025

MATH, released by Hendrycks et al. in 2021, is a dataset of 12,500 competition problems spanning five difficulty levels, and it defined math evaluation for years before frontier models saturated it (Source: Hendrycks et al., arXiv:2103.03874, 2021). AIME 2025 uses all 30 problems from the 2025 American Invitational Mathematics Examination, with integer answers from 000 to 999, and is tracked as a live leaderboard by Artificial Analysis (Source: Artificial Analysis, 2026). Because a fresh exam is released each year, comparing AIME 2023, 2024, and 2025 scores helps detect data contamination.

Which benchmarks measure knowledge and truthfulness?

Knowledge and truthfulness benchmarks test whether a model states facts correctly and resists confidently producing falsehoods, a different skill from reasoning. The leading examples are SimpleQA, for short-form factual accuracy, and TruthfulQA, for resistance to common misconceptions. Both also expose calibration, whether a model “knows what it knows,” because a well-calibrated model declines to answer when uncertain rather than hallucinating.

SimpleQA

SimpleQA is a factuality benchmark released by OpenAI in October 2024, authored by Wei et al. It contains 4,326 short, fact-seeking questions adversarially collected against GPT-4 so each has a single, indisputable answer (Source: OpenAI, 2024). Responses are graded as correct, incorrect, or not attempted. It proved hard even for strong models: OpenAI’s o1-preview reached only 42.7 percent at launch (Source: OpenAI, “Introducing SimpleQA,” 2024), making it a useful signal where older factuality tests had saturated.

TruthfulQA

TruthfulQA, authored by Lin et al. in 2022, takes the opposite angle. Its 817 adversarial questions across 38 categories deliberately target common misconceptions, conspiracies, and logical falsehoods, measuring whether a model repeats a plausible-sounding falsehood a human might believe (Source: Lin et al., arXiv:2109.07958, 2022). It has since saturated as its questions leaked into training data, which is exactly why newer factuality tests like SimpleQA were created.

Which benchmarks measure agentic behavior?

Agentic benchmarks test whether a model can use tools, follow rules, and complete multi-step tasks, not just answer a single prompt. They matter because real deployments rarely look like a quiz. The leading examples are tau-bench, for tool-agent-user conversations, and WebArena, for autonomous web navigation. Both are highly scaffold-dependent: tool access, retry budget, and evaluator version all materially change reported scores.

tau-bench

tau-bench (tool-agent-user benchmark) was developed by Sierra Research in 2024, authored by Yao et al. (Source: Sierra, arXiv:2406.12045, 2024). It simulates dynamic conversations between a user and an agent equipped with domain-specific API tools and policy rules, across retail and airline scenarios. Its standout contribution is the pass^k metric, which measures whether an agent completes a task consistently across repeated trials, not just once, exposing reliability gaps that single-shot tests miss.

WebArena

WebArena, introduced by Zhou et al. at Carnegie Mellon in 2023, runs agents inside fully functional self-hosted websites spanning e-commerce, social forums, and collaborative software, with 812 tasks generated from 241 templates (Source: Zhou et al., arXiv:2307.13854, 2023). It grades by objective string matching against task outcomes. Progress has been rapid: scores climbed from a 14.4 percent baseline toward roughly 60 percent by early 2025, still short of the 78.2 percent human benchmark.

Which benchmark measures human preference?

Human-preference benchmarks skip ground truth entirely and ask people which of two anonymous model responses they prefer. The dominant example is LMArena (formerly Chatbot Arena), which aggregates millions of blind pairwise votes into a single Elo-like rating. It is the closest public proxy for “which model feels better to use,” and the most-cited human-preference benchmark in AI (Source: LMArena, 2024).

LMArena (Chatbot Arena)

LMArena launched as Chatbot Arena from UC Berkeley in 2023, rebranded to LMArena in 2024, and spun out as Arena Intelligence in early 2026. It shows two unidentified models side by side, collects a vote, and converts votes using a Bradley-Terry model into ratings displayed in an Elo-like range; its methodology is documented in a peer-reviewed ICML 2024 paper (Source: Chiang et al., ICML 2024). By 2026 it had aggregated more than six million votes across over 400 models. Read the confidence intervals before the rank: top models routinely overlap, so their order is partly statistical noise.

Are there benchmarks for inference speed?

Yes, and this is the category most lists omit. Speed benchmarks measure how fast a model serves output, not how smart it is. The core metrics are output speed in tokens per second, time-to-first-token (TTFT), and inter-token latency. They determine whether an application feels instant or sluggish, and they vary enormously across providers serving the identical model (Source: Artificial Analysis, 2026).

Tokens per second and TTFT

Tokens per second measures the sustained generation rate; TTFT measures the delay before the first token appears; inter-token latency measures the gap between successive tokens. Artificial Analysis measures these under controlled workloads, testing 1k and 10k input-token prompts roughly every three hours and reporting median and p95 figures (Source: Artificial Analysis, 2026). The same model served by different providers can differ by an order of magnitude, which is why TokenDyno tracks live tokens-per-second across providers. A model that wins on accuracy can still lose in production if it generates text too slowly to feel responsive, so speed deserves first-class benchmark status. To rank models by capability first, see Which LLM Performs Best on Benchmarks?.

Frequently asked questions

What are examples of LLM benchmarks?

Examples include MMLU-Pro and GPQA Diamond (reasoning), SWE-bench Verified, HumanEval, and LiveCodeBench (coding), AIME and MATH (math), SimpleQA and TruthfulQA (knowledge), tau-bench and WebArena (agentic), LMArena (human preference), and tokens-per-second plus TTFT (speed). Each measures a distinct skill (Source: TIGER-Lab, 2024; Source: OpenAI, 2024).

What is the most important LLM benchmark?

There is no single most important benchmark, because each measures a different skill. The right one depends on your use case: SWE-bench Verified for coding agents, GPQA Diamond for hard reasoning, LMArena for general preference, and tokens-per-second for latency-sensitive products. Most teams weight two or three plus a private evaluation (Source: OpenAI, 2024).

Are there benchmarks for inference speed?

Yes. Speed benchmarks track tokens per second, time-to-first-token, and inter-token latency rather than accuracy. Independent measurers like Artificial Analysis run these under controlled workloads, and results vary by an order of magnitude across providers serving the same model, so speed must be evaluated separately from capability (Source: Artificial Analysis, 2026).

Why do benchmarks keep getting replaced?

Benchmarks get replaced when they saturate or are contaminated. Once frontier models all score in the low 90s, a test can no longer distinguish them, and leaked test questions inflate scores. MMLU, HumanEval, and TruthfulQA all saturated, triggering harder successors like MMLU-Pro, LiveCodeBench, and SimpleQA (Source: TIGER-Lab, 2024).

Who maintains these benchmarks?

Maintainers vary. Academic teams run MMLU (Hendrycks et al.), GPQA (NYU), and WebArena (Carnegie Mellon); OpenAI maintains SimpleQA and SWE-bench Verified; Sierra Research maintains tau-bench; and LMArena and Artificial Analysis run their respective live leaderboards. Each links a public source you can verify (Source: Rein et al., 2023; Source: Sierra, 2024).

Key takeaways

LLM benchmark examples fall into seven categories: reasoning (MMLU-Pro, GPQA Diamond), coding (SWE-bench Verified, HumanEval, LiveCodeBench), math (AIME, MATH), knowledge (SimpleQA, TruthfulQA), agentic (tau-bench, WebArena), human preference (LMArena), and speed (tokens-per-second, TTFT). No single test captures general capability, and many classic benchmarks have saturated, pushing the field toward harder, contamination-resistant successors.

When you read any benchmark, check who maintains it, whether it has saturated, and whether it tests tasks like yours. Then add the category most lists skip: measured inference speed. A model that scores well but serves slowly will still disappoint in production, so pair capability scores with live tokens-per-second before you ship.

Sources

← All posts