Alphabetized glossary index of LLM benchmarking terms laid out as catalog cards on white, one card highlighted in green

Blog · By Anton Gulin · Updated September 18, 2026

LLM Benchmarking Glossary: Every Term Explained

LLM benchmarking has two vocabularies that rarely sit in one place: a capability vocabulary (MMLU, pass@k, Elo, contamination) that grades how smart a model is, and a speed vocabulary (tokens/sec, TTFT, throughput, quantization) that grades how fast and cheaply it answers. This glossary defines both, accurately and with sources, so you can read any benchmark page without guessing.

Each term below gets a crisp definition and, where it maps to a specific paper or method, an inline citation. Terms are grouped into two categories, capability and quality terms first, then speed and efficiency terms, and alphabetized within each group. For the broader concept, see What Is an LLM Benchmark?; for worked examples, see Examples of LLM Benchmarks.

Capability and quality terms

These terms describe what a benchmark measures about a model’s intelligence, and how that measurement is graded. They cover named test suites (MMLU, GPQA), scoring methods (pass@k, Elo, LLM-as-judge), prompting setups (zero-shot, chain-of-thought), and the failure modes that quietly inflate or flatten scores (contamination, saturation).

Chain-of-thought (CoT)

A prompting technique that asks a model to produce intermediate reasoning steps before its final answer, rather than answering directly. Introduced by Wei et al. (2022), CoT was shown to sharply improve performance on arithmetic, symbolic, and commonsense reasoning tasks, especially in large models (Source: Wei et al., arXiv, 2022). Many reasoning benchmarks now assume CoT-style answering, which is why prompt format materially affects reported scores.

Contamination

Contamination occurs when benchmark test questions leak into a model’s training data, so the model recalls answers instead of reasoning them out, inflating its score. One audit estimated MMLU is roughly 29% contaminated, with one model dropping 13 points on a cleaned version of the test (Source: Pebblous, 2025). Contamination is the single biggest reason two different models can converge on suspiciously similar scores.

Elo rating (Bradley-Terry)

An Elo rating converts pairwise human preference votes into a single comparable strength score, originally a chess ranking method. LMArena (formerly Chatbot Arena) collects votes on anonymous head-to-head model outputs and fits a Bradley-Terry model, the maximum-likelihood form of Elo, to produce its leaderboard ratings (Source: Chiang et al., arXiv, 2024). It measures preference, not raw correctness.

GPQA

GPQA (Graduate-Level Google-Proof Q&A) is a multiple-choice benchmark of expert-written science questions designed so that even skilled non-experts with web access struggle to answer them. Introduced by Rein et al. (2023), its hardest subset, GPQA Diamond, keeps only questions that domain experts answer correctly and most non-experts miss (Source: Rein et al., arXiv, 2023). It targets genuine expert reasoning rather than retrievable facts.

LLM-as-judge

LLM-as-judge uses a strong language model to grade the outputs of other models, scoring open-ended answers that have no single correct response. Zheng et al. (2023) showed a GPT-4 judge can reach over 80% agreement with human preferences, comparable to human-human agreement, while documenting position, verbosity, and self-enhancement biases (Source: Zheng et al., arXiv, 2023). It is scalable but inherits the judge’s blind spots.

MMLU

MMLU (Massive Multitask Language Understanding) is a 4-option multiple-choice benchmark of roughly 14,000 questions across 57 subjects, from elementary math to professional law. Introduced by Hendrycks et al. (2021), it tests broad world knowledge and problem-solving in zero-shot and few-shot settings; human domain experts score about 89.8% (Source: Hendrycks et al., arXiv, 2021). It became the default capability yardstick, and is now largely saturated.

pass@k

pass@k measures functional correctness in code generation: a model samples k candidate solutions per problem, and the problem counts as solved if any one of them passes the held-out unit tests. Popularized by the Codex/HumanEval paper (Chen et al., 2021), it uses an unbiased estimator to reduce variance (Source: Chen et al., arXiv, 2021). pass@1 measures first-try success; higher k rewards models that get there with more attempts.

Saturation

Saturation is the point where top models cluster near a benchmark’s maximum score, so the test no longer separates them. MMLU, released in 2020, took years to saturate; GPQA Diamond reached around 94% on frontier models within roughly two years (Source: Stanford HAI AI Index, 2025). Saturation is why benchmarks are retired and replaced by harder successors on an accelerating cycle.

Zero-shot and few-shot

Zero-shot means a model is evaluated on a task with no worked examples in the prompt, only the question and instructions. Few-shot supplies a handful of solved examples first, which usually raises scores. The setting is part of the score: an “MMLU 88%” result means little until you know whether it was measured zero-shot or 5-shot, because the two are not comparable.

Speed and efficiency terms

These terms describe how fast and how cheaply a model serves answers, the axis that increasingly decides production choices once capability scores converge. They cover user-facing latency (TTFT, ITL, tokens/sec), system-level capacity (throughput, batch size), the two phases of inference (prefill, decode), and the main lever for trading quality against speed (quantization). For a deeper treatment, see LLM Tokens Per Second.

Batch size

Batch size is the number of requests an inference server processes together in one pass. Larger batches raise total throughput by sharing fixed GPU work across more requests, but can increase per-request latency. Batching is the central lever in the latency-versus-throughput trade-off: optimizing for one usually costs the other (Source: NVIDIA, 2025).

Decode

Decode is the second phase of inference, where the model generates output one token at a time after the prompt has been processed. It is typically memory-bandwidth-bound and sets the inter-token latency a user experiences while text streams (Source: NVIDIA, 2025). Decode speed, not prefill, dominates the wait for long responses.

Inter-token latency (ITL)

ITL is the average time between consecutive output tokens, capturing the model’s steady-state generation speed during the decode phase. It excludes the first token, so it characterizes only token-by-token generation (Source: NVIDIA, 2025). ITL is the inverse of the streaming tokens-per-second a user sees, and is sometimes called time per output token (TPOT).

Latency

Latency is the time a user waits for a response. End-to-end latency is the full duration from sending a prompt to receiving the complete answer, and equals time to first token plus total generation time (Source: NVIDIA, 2025). Latency-sensitive products like voice agents care most about TTFT; long-form generation cares most about ITL.

Prefill

Prefill is the first phase of inference, where the model reads the entire input prompt and populates its key-value (KV) cache before generating any output. It is usually compute-bound, and its duration grows with prompt length, which is why prefill effectively determines time to first token (Source: NVIDIA, 2025).

Quantization

Quantization reduces a model’s memory and compute cost by storing weights and activations in lower-precision formats such as INT8 or INT4 instead of FP16/FP32. INT8 makes a model roughly 4x smaller and INT4 roughly 8x smaller, speeding inference and cutting KV-cache size, usually with a small accuracy trade-off that grows at INT4 (Source: Hugging Face, 2025). It is the main lever for fitting larger models on smaller hardware.

Throughput

Throughput is the total number of tokens a system produces per unit of time across all concurrent requests, the key capacity metric for serving many users at once. It is measured in tokens per second at the system level and trades off against per-request latency (Source: NVIDIA, 2025). Higher throughput lowers cost per token, which is why providers optimize for it under load.

Time to first token (TTFT)

TTFT is the time from submitting a prompt to receiving the first output token. It includes request queuing, prefill, and network latency, and grows with prompt length because the model must process the whole input before generating (Source: NVIDIA, 2025). TTFT is the metric that decides whether an interactive experience feels instant or sluggish.

Tokens per second (tokens/sec)

Tokens per second is the rate at which a model generates output tokens, the headline speed metric for LLM inference. As a per-user streaming rate it reflects decode speed; as a system metric it reflects total throughput. Independent benchmarks like Artificial Analysis measure output tokens per second on live API endpoints (Source: Artificial Analysis, 2026). Provider rates for the same model can vary by an order of magnitude.

Frequently asked questions

What does tokens per second mean?

Tokens per second is the rate at which a model produces output tokens. As a per-user streaming rate it reflects how fast text appears on screen (decode speed); as a system metric it reflects total throughput across all requests. Independent benchmarks measure it on live API endpoints, where rates for the same model vary widely (Source: Artificial Analysis, 2026).

What is pass@k?

pass@k is a code-generation metric for functional correctness. A model samples k candidate solutions per problem, and the problem counts as solved if any one passes the held-out unit tests. Popularized by the Codex/HumanEval paper, it uses an unbiased estimator to reduce variance; pass@1 measures first-try success (Source: Chen et al., arXiv, 2021).

What is benchmark contamination?

Benchmark contamination is when test questions leak into a model’s training data, so it recalls answers instead of reasoning them out, inflating scores. One audit estimated MMLU is roughly 29% contaminated, with one model dropping 13 points on a cleaned test (Source: Pebblous, 2025). It can make distinct models look deceptively similar.

What is the difference between TTFT and ITL?

TTFT (time to first token) is the wait before the first output token appears, driven by prefill and prompt length. ITL (inter-token latency) is the average gap between subsequent tokens during streaming, driven by the decode phase. TTFT decides perceived responsiveness; ITL decides how fast long answers complete (Source: NVIDIA, 2025).

What does it mean when a benchmark is saturated?

A benchmark is saturated when top models cluster near its maximum score, so it no longer separates them. MMLU and GPQA Diamond both reached high-90s on frontier models within a few years (Source: Stanford HAI AI Index, 2025). Saturated benchmarks get retired and replaced by harder successors.

What is the difference between latency and throughput?

Latency is the time one user waits for a response; throughput is the total tokens a system produces per second across all users. They trade off: larger batch sizes raise throughput but can increase per-request latency. Interactive apps optimize for latency; high-volume serving optimizes for throughput (Source: NVIDIA, 2025).

Sources

← All posts