Seven distinct measuring instruments arranged in a comparison grid, each pointing at a different value, representing competing LLM leaderboards judged side by side

Blog · By Anton Gulin · Updated June 28, 2026

LLM Leaderboards Compared: Which Ranking Should You Trust?

Last updated: 2026-06-28

There is no single LLM leaderboard, and the seven that matter most rarely agree. LMArena measures human preference, Artificial Analysis combines intelligence with speed and price, LiveBench limits contamination, HELM prioritizes transparency, SWE-bench grades real coding, Vellum curates fresh benchmarks, and the Hugging Face Open LLM Leaderboard is now archived. This comparison breaks down each one by method, what it is good and bad at, how gameable it is, how often it updates, and which question it actually answers — so you trust the right ranking for your decision instead of the loudest headline. It also helps explain why LLM leaderboard searches land on aggregators rather than on any single lab’s own numbers.

How do the major LLM leaderboards compare at a glance?

Seven leaderboards dominate model selection in 2026, and each scores a different thing: human preference, composite intelligence, contamination-limited capability, transparency, real coding, or curated freshness. Because their methods diverge, they crown different winners, so the right choice depends entirely on the question you are trying to answer.

The matrix below maps each leaderboard to its method, its best use, its update cadence, and its source. Frontier standings reshuffle within weeks, so treat this as a map of the landscape and click through for live numbers (accessed 2026-06-28).

LeaderboardMethodBest forUpdate cadenceSource
LMArena (now Arena)Anonymous A/B human votes → Bradley-Terry (Elo-style) rating”Which model feels best in open-ended chat”Continuous; new models added on releaselmarena.ai
Artificial AnalysisComposite Intelligence Index + independently measured speed, latency, priceBuyer’s full view: capability vs speed vs costContinuous; speed sampled daily, 100k workloads weeklyartificialanalysis.ai
LiveBenchObjective ground-truth scoring on monthly-refreshed questions, no LLM judgeContamination-resistant general capabilityMonthly additions; full refresh every 6 monthslivebench.ai
Stanford HELMStandardized scenarios scored on multiple metrics, full reproducibilityTransparency and auditable methodologyPeriodic; entered maintenance mode mid-2026crfm.stanford.edu/helm
SWE-benchGenerated patches graded by each repo’s own unit testsReal-world software engineeringContinuous submissions; Verified set fixed at 500 tasksswebench.com
VellumCurated public benchmark results on non-saturated tests, plus price and speedPractitioner shortlist on fresh benchmarksUpdated as models and benchmark results shipvellum.ai/llm-leaderboard
Hugging Face Open LLMFixed automated suite (IFEval, BBH, MATH, GPQA, MuSR, MMLU-Pro)Historical open-weight comparisons onlyArchived March 2025; no new submissionshuggingface.co (archived)

Why don’t the leaderboards agree on a single best model?

They disagree because they measure different abilities with different grading. A human-preference winner can write weaker code than the SWE-bench leader, and a reasoning champion can serve too slowly to ship. Artificial Analysis publishes intelligence, speed, and price as separate axes precisely because no one number captures a model’s fitness (Source: Artificial Analysis, 2026). When you see two leaderboards crown different models, that is the system working as designed, not a contradiction. The fix is to pick the axis that matches your workload, then read the leaderboard built for it.

What is the LMArena (Arena) leaderboard good and bad at?

LMArena is the largest human-preference leaderboard, having collected more than 3.5 million head-to-head votes across over 400 models as of September 2025 (Source: LMArena, 2025). Users vote between two anonymous responses, and votes feed a Bradley-Terry model displayed on an Elo-style scale anchored at 1500. It is the best proxy for which model people prefer in open-ended use.

How does LMArena’s method actually work?

Chatbot Arena originally used online Elo, then switched to Bradley-Terry in December 2023 for more stable estimation; the “Elo” label persists only by habit (Source: Latent Space / LMArena, 2024). Beating a strong opponent moves your score more than beating a weak one. In August 2024 the team added Style Control, regression terms that subtract the effect of response length, markdown headers, bold text, and lists, isolating substance from presentation (Source: LMArena, 2024). The platform rebranded to Arena and spun out as Arena Intelligence Inc. in January 2026.

What is LMArena bad at?

LMArena’s weakness is statistical and structural. The top several models routinely sit inside overlapping 95% confidence intervals, computed by bootstrap resampling the vote pool 1,000 times, so exact rank order is partly noise (Source: LMArena, 2026). Its voter pool skews toward AI enthusiasts. It also measures preference, not correctness: a confident, well-formatted wrong answer can still win a vote. Use it to gauge feel, not factual accuracy or coding ability.

What is Artificial Analysis good and bad at?

Artificial Analysis is the leaderboard closest to a full buying decision, because it ranks intelligence, output speed, latency, and price side by side instead of in isolation. Its Intelligence Index reached v4.1 in 2026, combining evaluations such as Humanity’s Last Exam, GPQA Diamond, Terminal-Bench, SciCode, and AA-LCR, all run independently by Artificial Analysis (Source: Artificial Analysis, 2026).

How does Artificial Analysis measure speed and intelligence?

Intelligence is a composite of independently run benchmarks. Speed is measured separately: output tokens per second is sampled with 10 concurrent requests on a standard 1,000-token input workload sent once per day at a random time, with a 100,000-token workload tested weekly, each run using a unique prompt (Source: Artificial Analysis, performance methodology, 2026). Because the same model can run several times faster on one provider than another, the speed view ranks model-and-provider combinations, not just models. For a focused three-way breakdown, see Livebench vs Artificial Analysis vs TokenDyno.

What is Artificial Analysis bad at?

Its composite Intelligence Index is a hidden editorial choice. Collapsing many benchmarks into one number means the weighting, not the models, can decide the order, and two leaderboards weighting coding versus reasoning differently will rank the same models differently (Source: Artificial Analysis, methodology, 2026). It is also less granular for human-feel and open-ended quality than a preference leaderboard. Read the underlying benchmark breakdown before trusting the single headline score.

What is LiveBench good and bad at?

LiveBench is the leading contamination-limited capability leaderboard, built by Abacus.AI with collaborators from NYU, Nvidia, and others and presented as an ICLR 2025 Spotlight. It currently spans 23 tasks across 7 categories — reasoning, coding, agentic coding, mathematics, data analysis, language, and instruction following — and is hard by design, with top models scoring below 70% (Source: LiveBench, 2026).

Why is LiveBench harder to game than most leaderboards?

LiveBench attacks contamination two ways. It adds new questions monthly and fully refreshes every six months, drawing from recent arXiv papers, news, and datasets so models cannot have trained on the test set (Source: LiveBench, 2026). The latest release at the time of writing was LiveBench-2026-01-08. It also scores every question against objective ground truth with no LLM judge, a choice made after the team found LLM-judge grading of hard reasoning and math hit error rates as high as 46% (Source: White et al., arXiv:2406.19314, 2025).

What is LiveBench bad at?

LiveBench measures capability, not deployment economics. It says nothing about serving speed, latency, or cost, and its objective-scoring requirement means it favors tasks with verifiable answers over open-ended quality and conversational nuance. Its monthly refresh also makes historical comparisons across versions less clean. Pair it with a speed-and-price source before committing to production.

What are HELM and the Hugging Face leaderboard good and bad at?

HELM and the Hugging Face Open LLM Leaderboard represent the academic, transparency-first tradition. HELM, from Stanford’s Center for Research on Foundation Models, prioritizes reproducibility; the Hugging Face board was the open-weight default until it was archived in March 2025 (Source: Hugging Face, 2025). Both are rigorous, but one is slowing and the other is frozen.

What does HELM do better than commercial leaderboards?

HELM’s edge is transparency. Its 2022 framework evaluated models across 42 scenarios on seven metric categories including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency (Source: Liang et al., arXiv:2211.09110, 2022). In March 2025 CRFM launched HELM Capabilities with full prompt-level reproducibility, and the project spans specialized boards like MedHELM and AIR-Bench for safety (Source: Stanford CRFM, 2025). When you need to audit exactly how a score was produced, HELM is the most open option. It entered maintenance mode in mid-2026, so it updates more slowly than commercial rivals.

Why is the Hugging Face Open LLM Leaderboard only useful as history?

The Hugging Face board scored open-weight models on a fixed suite — IFEval, BBH, MATH, GPQA, MuSR, and MMLU-Pro — but it was archived in March 2025 and no longer accepts submissions, with maintainers pointing users to contamination-resistant alternatives (Source: Hugging Face, 2025). It cannot reflect any model released since, so its standings age daily. Treat it as a historical reference for older open-weight models, not a current ranking. For broader navigation of the landscape, see LLM Leaderboard 2026.

What are SWE-bench and Vellum good and bad at?

SWE-bench and Vellum are the most practitioner-facing leaderboards. SWE-bench grades models on real GitHub issues by running each repository’s own unit tests, so a patch passes or fails objectively (Source: Jimenez et al., arXiv:2310.06770, 2023). Vellum curates published benchmark results on deliberately non-saturated tests, giving a fast practitioner shortlist with price and speed attached.

Why is SWE-bench Verified being deprecated, and what replaces it?

SWE-bench Verified is a 500-task subset that experts validated as solvable, but by 2026 OpenAI announced it would no longer evaluate on it, citing contamination and flawed tasks (Source: OpenAI, 2026). Their audit found that 59.4% of 138 hard problems had material test or description issues, and frontier models could reproduce verbatim gold patches. Scale AI’s SWE-bench Pro, released August 2025, counters this with GPL-copyleft and private repositories as a legal contamination deterrent; top models score roughly 23% on Pro versus 70%-plus on Verified (Source: Scale AI, 2026). Scaffold choice also swings SWE-bench scores heavily.

What is Vellum’s leaderboard good for, and what is its limitation?

Vellum’s strength is curation and freshness. It tracks models released after April 2024 on non-saturated benchmarks — GPQA Diamond, AIME 2025, SWE-bench, ARC-AGI-2, Humanity’s Last Exam, and MMMLU — explicitly excluding saturated tests like MMLU, with a separate coding board using LiveCodeBench, Aider Polyglot, and BFCL (Source: Vellum, 2026). Its limitation is that much of its data is reported by model providers rather than independently re-run, so it inherits any optimism in vendor-published numbers. Use it as a starting shortlist, then verify on independently run sources. For the tests themselves, see LLM Benchmark: A Complete Guide.

Which LLM leaderboards are easiest to game?

Human-preference leaderboards are the most gameable, capability leaderboards with private test sets and objective grading are the hardest. The gap is structural: anything scored by crowd votes or static public questions can be optimized against, while contamination-limited and unit-test-graded designs resist tuning. No leaderboard is immune, but the methods differ sharply in exposure.

How was Chatbot Arena shown to be gameable?

Two peer-reviewed findings exposed Arena’s exposure. “The Leaderboard Illusion” (NeurIPS 2025) analyzed roughly 2 million battles across 243 models and reported that a few large providers got disproportionate data access and could privately test many variants — identifying up to 27 private Meta variants before one launch — before publishing only their best (Source: Singh et al., arXiv:2504.20879, 2025). Separately, an ICML 2025 paper demonstrated that crowdsourced votes can be rigged to move a target model’s rank, even via battles it is not in (Source: ICML, 2025). LMArena disputed several figures and points to countermeasures like style control and bot detection (Source: LMArena, 2025).

Why are contamination-limited leaderboards harder to game?

Objective grading and fresh questions remove the two easiest attack surfaces. LiveBench’s monthly question rotation means there is no stable test set to overfit, and its no-LLM-judge scoring removes judge manipulation (Source: LiveBench, 2026). SWE-bench Pro’s copyleft and private repos act as a legal and practical barrier to training-data leakage (Source: Scale AI, 2026). These designs do not eliminate gaming, but they raise its cost. The lesson: weight leaderboards whose test sets are fresh or privately held over those whose questions are public and static.

How often is each LLM leaderboard updated?

Update cadence ranges from continuous to frozen, and it directly affects trust. LMArena and Artificial Analysis update continuously as models ship; LiveBench refreshes monthly with a full six-month rotation; SWE-bench takes rolling submissions; HELM updates periodically and slowed in 2026; and the Hugging Face Open LLM Leaderboard has been frozen since March 2025.

Why does update cadence determine trustworthiness?

A leaderboard is only useful if its standings move when models do. Frontier releases land roughly monthly, so a ranking written into a blog post or a frozen table is often stale within weeks. The archived Hugging Face board is the cautionary case: useful as history, but it cannot reflect any model released after March 2025 (Source: Hugging Face, 2025). Prefer continuously updated sources, and always record the date you accessed a leaderboard. A rank without an access date is a claim without a timestamp.

Which LLM leaderboard should you trust for your question?

Match the leaderboard to the decision. For conversational feel, trust LMArena; for a combined capability-speed-cost view, trust Artificial Analysis; for contamination-resistant capability, trust LiveBench; for coding, trust SWE-bench and its successors; for transparency, trust HELM; and for a fast practitioner shortlist, trust Vellum. No single board answers all four questions. These are also the boards search engines lean on, which is how AI Overviews cite LLM leaderboards when summarizing which model is best.

Which leaderboard is best for capability and reasoning?

For raw capability, prefer LiveBench and HELM Capabilities over saturated or preference-based boards. LiveBench’s monthly-refreshed, objectively graded questions resist contamination, and top models still score below 70%, leaving headroom to separate them (Source: LiveBench, 2026). HELM adds reproducibility when you need to audit the method. Cross-check both against Vellum’s non-saturated benchmark set, but verify any vendor-reported figure on an independently run source before deciding.

Which leaderboard is best for coding?

For coding, SWE-bench remains the reference because it grades real patches with real unit tests, but read it carefully in 2026. SWE-bench Verified is being deprecated for contamination, so weight contamination-resistant successors like SWE-bench Pro and remember that scores swing with the agent scaffold (Source: OpenAI, 2026; Scale AI, 2026). Vellum’s coding board, built on LiveCodeBench and Aider Polyglot, is a useful cross-check on practical code editing.

Which leaderboard is best for speed and cost?

For speed and cost, capability leaderboards are nearly silent — only Artificial Analysis treats serving speed and price as first-class axes, measuring output tokens per second and latency on a fixed daily cadence (Source: Artificial Analysis, 2026). Because the same model runs at very different speeds across providers, speed rankings cover model-and-provider pairs. A live tokens-per-second view complements the accuracy boards above when latency decides whether your application feels instant or sluggish.

Frequently asked questions

Which LLM leaderboard is most accurate?

No single leaderboard is most accurate, because each measures a different thing. For contamination-resistant capability, LiveBench is among the most reliable thanks to monthly-refreshed questions and objective grading. For a combined intelligence, speed, and price view, Artificial Analysis is the most complete. Match the board to your workload (Source: LiveBench, 2026).

Is LMArena reliable?

LMArena is reliable as a measure of human preference, with over 3.5 million votes across 400-plus models, but it has known limits. Top models share overlapping confidence intervals, its voter pool skews technical, and peer-reviewed work shows it can be gamed via private testing and vote manipulation. Treat it as feel, not correctness (Source: Singh et al., 2025).

Which leaderboard is best for coding?

SWE-bench is the strongest coding leaderboard because it grades real GitHub patches with repository unit tests. In 2026, prefer contamination-resistant versions like SWE-bench Pro, since SWE-bench Verified is being deprecated for contamination, and note that scores depend heavily on the agent scaffold used (Source: OpenAI, 2026).

Which leaderboard is best for speed?

For speed, Artificial Analysis is the main capability-grade source, measuring output tokens per second and latency on a fixed cadence. Because the same model runs at very different speeds across providers, dedicated speed views rank model-and-provider combinations, so evaluate serving speed separately from accuracy before deployment (Source: Artificial Analysis, 2026).

Why do different LLM leaderboards show different winners?

Different leaderboards reward different skills, so they crown different leaders. Human-preference rankings, coding tests, and reasoning benchmarks rarely agree, because each uses a distinct grading method. A reasoning champion may write weaker code or serve slowly, so consult the leaderboard whose metric matches your use case (Source: Artificial Analysis, 2026).

Can LLM leaderboards be trusted at all?

Yes, when read correctly. Trust a rank most when the test set is fresh, the grading is objective, the leaderboard updates continuously, and the metric matches your task. Read confidence intervals before ranks, discount saturated benchmarks, and cross-check vendor-reported numbers against independently run sources (Source: LiveBench, 2026).

Key takeaways

No single LLM leaderboard is definitive, and the seven that matter answer different questions. LMArena measures human preference but shares overlapping confidence intervals and is gameable; Artificial Analysis is the most complete buyer’s view across intelligence, speed, and price; LiveBench is the most contamination-resistant capability board; HELM is the most transparent; SWE-bench is the coding reference, though Verified is being deprecated for contamination; Vellum is a fast practitioner shortlist on fresh benchmarks; and the Hugging Face Open LLM Leaderboard is now a frozen historical reference.

The decision rule is simple: match the leaderboard to your question, prefer fresh and objectively graded sources over static and preference-based ones, read confidence intervals before ranks, and always pair a capability ranking with a live speed-and-cost view before you commit. Frontier standings reshuffle within weeks, so revisit them with a dated access note rather than trusting a number you read once.

Sources

← All posts