Grid of coding benchmark panels on white comparing AI models, each panel a terminal window with a green pass-rate bar climbing a chart

Blog · By Anton Gulin · Updated June 28, 2026

LLM Coding Benchmark: How Models Compare on Code

Last updated: 2026-06-28

An LLM coding benchmark measures how well a model writes, edits, or debugs code by running its output against unit tests. The leading suites, SWE-bench, HumanEval, MBPP, LiveCodeBench, BigCodeBench, Aider polyglot, and Terminal-Bench, each test a different skill, from single-function generation to multi-file agentic repair (Source: SWE-bench, 2024).

No single number captures coding ability. A model can ace 164 algorithm puzzles yet fail to fix one real GitHub issue. This guide explains what each benchmark tests, how pass@k scoring works, why contamination distorts results, and where to read current standings. Every figure below traces to a named 2024-2026 source you can check.

What is an LLM coding benchmark?

An LLM coding benchmark is a fixed set of programming tasks with hidden unit tests. The model generates a solution, the harness executes it, and the score is the share of tasks whose tests pass (Source: OpenAI, 2021). It grades functional correctness, not how the code reads or how fast it runs, which is why subjective qualities are pushed to separate tests like an LLM creative-writing benchmark that scores prose quality instead of pass rates.

Coding benchmarks split along scope. Function-level suites like HumanEval hand the model one docstring and ask for one function. Repository-level suites like SWE-bench drop the model into a real codebase and a real bug report, then check whether its patch makes the failing tests pass (Source: SWE-bench, 2024). The first measures raw code generation; the second measures whether a model can act as a software engineer. Both report execution-based scores, which is what separates modern coding benchmarks from older string-matching metrics like BLEU.

What does pass@k actually measure?

Pass@k is the probability that at least one of k generated samples passes every unit test for a problem. The model produces k independent attempts; the problem counts as solved if any one of them is fully correct (Source: OpenAI, 2021). Pass@1, the strictest and most cited variant, usually uses greedy decoding for a deterministic single attempt.

The k matters more than people assume. Pass@10 and pass@100 let a model “buy” correctness with volume, so they flatter weaker models that occasionally stumble onto the answer. Pass@1 reflects what a user feels on the first try. When you compare two scores, confirm they use the same k and the same decoding settings. A pass@10 figure is not comparable to a pass@1 figure, even on the identical benchmark.

How does SWE-bench test real-world coding?

SWE-bench gives a model a real GitHub issue and the surrounding codebase, then asks it to generate a patch. The harness applies the patch and runs the repository’s test suite; the task passes only if the previously failing tests now pass. The full set holds 2,294 instances mined from 12 Python projects (Source: SWE-bench, 2024).

This design is why SWE-bench became the headline coding benchmark. It rewards skills no function-level test touches: reading an unfamiliar codebase, locating the right file, and editing across modules without breaking existing tests. Scores climbed fast as agentic scaffolds matured, but the gap between subsets is large. Claude Opus 4.5 scores about 80.9% on SWE-bench Verified yet drops to roughly 45.9% on the harder, contamination-resistant SWE-bench Pro (Source: morphLLM, 2026). The same model, the same kind of task, a 35-point fall.

SWE-bench Verified vs SWE-bench Full

SWE-bench Verified is a 500-task subset that OpenAI and the SWE-bench authors hand-filtered for solvability. Human annotators reviewed 1,699 problems, three experts each, removing underspecified statements and overly strict tests; 68.3% of the original samples were filtered out (Source: OpenAI, 2024). Verified is the version most model cards report.

The cleanup mattered because the full set was noisy. Annotators flagged 38.3% of samples for underspecified problems and 61.1% for unit tests that could fail valid solutions (Source: OpenAI, 2024). Verified trades breadth for trust: every task is confirmed solvable with the information given. Use Verified for cross-model comparison, but remember it is now mature, widely trained on, and saturating at the top, which is exactly why harder variants like SWE-bench Pro appeared.

What do HumanEval and MBPP test?

HumanEval and MBPP are the original function-level Python benchmarks. HumanEval, released by OpenAI in 2021, holds 164 hand-written problems, each a function signature, a docstring, and hidden tests. MBPP adds 974 “mostly basic” programming problems covering everyday Python (Source: OpenAI, 2021).

Both are largely saturated. Top models now score above 99% on HumanEval, so the benchmark no longer separates frontier systems; it survives mainly as a sanity check and a historical baseline (Source: Statology, 2025). The EvalPlus project extended them into HumanEval+ and MBPP+ by adding tougher hidden tests and fixing wrong reference solutions, which exposed models that had memorized the visible cases. Treat a near-perfect HumanEval score as table stakes, not as evidence of strong engineering ability. It tells you a model can write a correct sort function, nothing about whether it can fix a bug in a 50,000-line repo.

How do LiveCodeBench and BigCodeBench differ?

LiveCodeBench and BigCodeBench both answer weaknesses in HumanEval, but in opposite directions. LiveCodeBench fights contamination by continuously adding fresh competitive-programming problems with known release dates. BigCodeBench fights triviality by demanding realistic, library-heavy code (Source: Jain et al., 2025; BigCode, 2024).

LiveCodeBench, from UC Berkeley, scrapes new problems from contest sites and tags each with its publication date, so you can score a model only on problems released after its training cutoff, the cleanest defense against memorization. Version 6 holds 1,055 problems spanning May 2023 to April 2025, and frontier reasoning models pass roughly 80% on recent windows (Source: Jain et al., 2025). BigCodeBench takes a different angle: its tasks require calling diverse functions from real libraries to satisfy complex instructions, using calibrated pass@1 with greedy decoding (Source: BigCode, 2024). Models that crush algorithm puzzles often fumble these practical API-fluency tasks, which is precisely the gap it was built to expose.

Why does contamination resistance matter so much?

Contamination resistance matters because most coding benchmarks are public, so their problems and solutions leak into training data. A model that memorized the answer looks competent without being able to generalize. LiveCodeBench’s date filtering is the main structural fix in current use (Source: Jain et al., 2025).

The effect is measurable. When a benchmark is fresh, scores reflect reasoning; once it has circulated for a year or two, scores partly reflect recall. That is why a model can post a sky-high SWE-bench Verified number while OpenAI’s own audit found that all frontier models could reproduce the verbatim gold patch or problem details, a signature of exposure (Source: OpenAI, 2026). Always check whether a benchmark controls for the model’s training cutoff before reading its scores as pure capability.

What do Aider polyglot and Terminal-Bench measure?

Aider polyglot and Terminal-Bench measure applied, multi-step coding rather than isolated generation. Aider polyglot tests code editing across six languages; Terminal-Bench tests whether an agent can complete end-to-end tasks in a real command line (Source: Aider, 2026; Terminal-Bench, 2025).

Aider polyglot runs 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust. The model gets two attempts, receiving the test error after the first, so it scores both initial skill and the ability to repair code from feedback and to emit a correctly formatted diff (Source: Aider, 2026). Terminal-Bench, a Stanford and Laude Institute project, drops agents into containerized terminal environments for tasks like compiling code, training models, and configuring systems. Its 2.0 release (November 2025) holds 89 human-verified tasks, and frontier agents still score under 65% (Source: Terminal-Bench, 2025). These two sit closest to how engineers actually use models day to day.

Which coding benchmark measures what?

The seven benchmarks below cover a ladder from single functions to full agentic workflows. Reading across the table shows why one score never tells the whole story: each suite isolates a different skill and reports it with a different scoring convention.

BenchmarkWhat it testsFormat / scoringSource
HumanEvalSingle-function Python generation from a docstringpass@k (usually pass@1, greedy); 164 problemsOpenAI, 2021
MBPPBasic everyday Python functionspass@k; 974 problemsOpenAI, 2021
BigCodeBenchRealistic tasks using diverse library callsCalibrated pass@1, greedyBigCode, 2024
LiveCodeBenchFresh competitive-programming problems (contamination-controlled)pass@1, date-windowed; v6 1,055 problemsJain et al., 2025
Aider polyglotCode editing across 6 languages with error feedbackPass rate over 225 Exercism tasks, 2 attemptsAider, 2026
SWE-bench VerifiedPatching real GitHub issues in a codebase% resolved; 500 human-validated tasksOpenAI, 2024
Terminal-Bench 2.0End-to-end agentic tasks in a real terminal% task completion; 89 tasksTerminal-Bench, 2025

Read the table top to bottom and the difficulty climbs with realism. HumanEval and MBPP grade isolated functions and are saturated. BigCodeBench and LiveCodeBench add library complexity and contamination control. Aider polyglot, SWE-bench, and Terminal-Bench grade applied engineering, where frontier models still leave real headroom. Match the benchmark to your use case: a code-completion tool cares about LiveCodeBench, an autonomous fix-it agent cares about SWE-bench and Terminal-Bench. For a deeper catalog, see Examples of LLM Benchmarks.

How are coding benchmarks gamed or contaminated?

Coding benchmarks are gamed mainly through contamination and scaffold tuning. Contamination happens when public test problems leak into training data, so the model recalls rather than reasons. Scaffold tuning happens when a vendor wraps the model in a custom agent built to maximize one benchmark (Source: OpenAI, 2026).

Both inflate scores without proving general ability. OpenAI stopped reporting SWE-bench Verified in early 2026, noting that residual underspecified tasks and clear contamination signals made the number unreliable for frontier comparison (Source: OpenAI, 2026). Self-reporting compounds the problem: many leaderboards mix vendor-submitted figures with independently run ones, and harness differences alone can move a score several points (Source: Aider, 2026). The defenses are date-windowed problems (LiveCodeBench), human-validated tasks (SWE-bench Verified), neutral scaffolds (Terminal-Bench’s Terminus 2), and harder successor sets (SWE-bench Pro). When you read any score, ask three questions: Is it self-reported? What scaffold produced it? Could the model have trained on the problems?

Why does a fast coding model still need to be correct?

A fast model is worthless if its code fails the tests. Speed and correctness are independent axes: throughput, measured in tokens per second, says nothing about whether a patch resolves the issue. A model that streams 800 tokens per second but fails SWE-bench wastes both its speed and your time (Source: SWE-bench, 2024).

The two axes interact in practice, though. Reasoning models that “think” before answering tend to score higher on hard coding tasks but generate slower, while small fast models stream quickly and miss more edge cases. The right pick depends on the job: a high-volume code-completion feature can favor a fast model with a good-enough LiveCodeBench score, while an autonomous bug-fixing agent should favor SWE-bench correctness even at lower speed. TokenDyno tracks the speed axis, live tokens-per-second across providers, so you can weigh it against the correctness scores these coding benchmarks report. For the capability side, see Which LLM Performs Best on Benchmarks?.

Which models lead the coding benchmarks now?

Standings change weekly, so read them on live boards rather than trusting any fixed figure. As of 2026-06-28, the SWE-bench Verified leaderboard shows top self-reported scores clustering in the high-80s to mid-90s percent, while the official Aider polyglot board lists GPT-5 (high) near 88% with Anthropic’s Claude Opus 4.5 self-reporting about 89.4% (Source: Steel.dev, 2026; Aider, 2026).

Treat those numbers as directional, not precise. The Aider polyglot leaderboard separates officially re-run results from vendor-submitted ones, and frontier models released after its last run may not appear yet (Source: Aider, 2026). On LiveCodeBench, frontier reasoning models pass roughly 80% on recent date windows, and the open-weight leaders trail proprietary models by a meaningful margin (Source: Jain et al., 2025). For the canonical sources, check the SWE-bench leaderboard at swebench.com, the Aider board at aider.chat/docs/leaderboards, and LiveCodeBench at livecodebench.github.io, each with your access date noted. For a curated current view, see the LLM Coding Benchmark Leaderboard.

Frequently asked questions

What is the best LLM for coding?

There is no single best; it depends on the task and which benchmark you weight. As of 2026-06-28, Claude and GPT-5 models trade the top of SWE-bench Verified and Aider polyglot, with leading agentic scores in the high-80s to mid-90s percent, but these are largely self-reported (Source: Steel.dev, 2026; Aider, 2026). Check live boards before deciding.

What is SWE-bench?

SWE-bench is a coding benchmark that gives a model a real GitHub issue and codebase, then checks whether its generated patch makes the failing tests pass. The full set has 2,294 tasks; the human-validated Verified subset has 500 (Source: SWE-bench, 2024; OpenAI, 2024). It measures real software-engineering ability, not just function writing.

How are coding benchmarks gamed or contaminated?

Mainly two ways: contamination, where public test problems leak into training data so the model recalls answers, and scaffold tuning, where a vendor builds a custom agent to maximize one benchmark. OpenAI dropped SWE-bench Verified in 2026 partly over contamination signals (Source: OpenAI, 2026). Date-windowed and human-validated benchmarks reduce both.

What is pass@k in coding benchmarks?

Pass@k is the probability that at least one of k generated solutions passes all unit tests for a problem. Pass@1, the strictest variant, allows one attempt and usually uses greedy decoding (Source: OpenAI, 2021). Higher k flatters weaker models, so always compare scores at the same k.

Why are HumanEval scores no longer useful?

HumanEval is saturated. Top models score above 99% on its 164 problems, so it no longer separates frontier systems (Source: Statology, 2025). It survives as a sanity check. Harder successors like BigCodeBench, LiveCodeBench, and SWE-bench now do the discriminating work for capable models.

Do coding benchmarks measure speed?

No. Coding benchmarks measure correctness, the share of tasks whose tests pass, not how fast a model generates tokens. Speed is a separate axis tracked by inference benchmarks in tokens per second (Source: SWE-bench, 2024). A correct-but-slow model and a fast-but-wrong model can post identical correctness scores only by coincidence.

Sources

← All posts