A model ranking table where every row carries both a capability bar and a green speed gauge, showing speed and intelligence as co-equal axes

Blog · By Anton Gulin · Updated August 21, 2026

Why Tokens/sec Belongs in Every LLM Ranking

Most LLM leaderboards still rank models on a single axis: how smart is it. That made sense when a model produced one answer to one prompt for one human who was happy to wait. It no longer matches how the systems are built. Models now think out loud for tens of seconds, agents chain dozens of calls into a single task, and the user is often another program waiting on a deadline. In that world, output speed in tokens per second and time to first token (TTFT) are not a footnote to capability; they are a co-equal axis. This is the argument, with sources, for why every ranking should carry a speed column.

Why should tokens/sec be a first-class ranking axis?

Because a model’s usefulness is throughput times quality, not quality alone. Artificial Analysis tracks output speed (tokens per second) and time to first token as primary metrics precisely because the same model served by different providers can swing from roughly 51 to 67 tokens per second, a gap large enough to change which model you should ship (Source: Artificial Analysis, 2026).

A capability-only ranking implicitly assumes speed is free or constant. It is neither. Two models can sit one row apart on a quality board and differ by an order of magnitude in delivery rate, which means the “second-best” model can finish a task while the “best” one is still emitting its reasoning trace. When the consumer of the output is an agent on a timeout, the slower model has effectively failed the task regardless of its score. Ranking on quality alone hides that failure mode entirely.

What does inference speed actually measure?

Inference speed is not one number; it is three. TTFT measures the seconds from request to the first token, output speed measures tokens per second during generation, and end-to-end response time combines input processing, reasoning, and answer generation into the total a user waits (Source: Artificial Analysis, 2026). Each maps to a different part of the experience.

TTFT defines the first impression: it is the moment a streaming interface stops feeling frozen (Source: IBM, 2025). Output speed governs how fast a long answer fills in once it starts. End-to-end latency is what a non-streaming caller, like an agent waiting for a complete JSON object, actually experiences. End-to-end latency equals time to first token plus total generation time, so a fast per-token rate cannot rescue a model that spends 40 seconds thinking before the first token appears (Source: Anyscale, 2025). A ranking that reports only one of these tells a partial story.

Why is TTFT sensitive to where you measure it?

TTFT includes network round-trip time, so it depends on server location, not just the model. Artificial Analysis runs its primary tests from Google Cloud’s us-central1-a zone and notes this can advantage or disadvantage providers depending on where they host (Source: Artificial Analysis, 2026).

This is why speed has to be measured, not assumed. A model that posts a fast TTFT from one region can look sluggish from another, and a provider’s serving stack, batching, speculative decoding, and hardware all move the number independently of the weights. Output speed also varies with the type of output because techniques like speculative decoding accelerate some token sequences more than others (Source: Artificial Analysis, 2026). Speed is a property of the deployed system, which is exactly why it belongs on a live board rather than in a model’s spec sheet.

Why do reasoning models burn more tokens and feel slow?

Because they generate a hidden monologue before answering. Reasoning models produce extensive internal chain-of-thought tokens before any visible output, and on GPT-5-class systems, pushing reasoning effort to its maximum has been measured to raise token usage and cost by roughly four to five times while improving accuracy by only a few points (Source: Towards Data Science, 2025).

That trade is the heart of the speed problem. The latency a user feels from a reasoning model is dominated by the thinking phase, which can run from ten to over a hundred seconds, so the slowdown shows up as end-to-end response time rather than per-token rate (Source: Artificial Analysis, 2026). The trend is structural, not a quirk of one model. Gartner projects that frontier intelligence will demand far more tokens than today’s mainstream apps, with agentic models requiring five to thirty times more tokens per task than a standard chatbot (Source: Gartner, 2026). More tokens at a fixed token rate means more wall-clock time, every single call.

How does latency compound in agentic workflows?

It compounds multiplicatively, because an agent does not make one call; it makes many. Agentic workflows are goal-driven sequences in which the model is invoked repeatedly to plan, act, and recover from failures, and each reasoning step adds inference cost and end-to-end latency that can become prohibitive in user-facing applications (Source: arXiv, 2026). The math is unforgiving.

Consider a concrete pattern. A four-agent debate run for five rounds is at least twenty LLM calls, each carrying the full accumulated conversation history (Source: Gurusup, 2026). If each call has a one-second TTFT and generates a 400-token response, then at 60 tokens per second each step costs roughly 7.7 seconds and the task takes about 154 seconds; at 600 tokens per second each step costs about 1.7 seconds and the same task finishes in roughly 34 seconds. The model is identical in capability terms, but a 10x difference in output speed is the difference between a 2.5-minute wait and a half-minute one. Speed does not add across steps; it multiplies.

Why does adding more agents make this worse?

Because coordination overhead grows faster than the benefit. When a single agent already exceeds about 45% accuracy, adding more agents often yields diminishing or even negative returns, and in tool-heavy setups with more than roughly ten APIs, distributed agent systems can suffer two-to-six-times efficiency losses from context fragmentation (Source: Sesame Disk, 2026).

Every additional agent is another chain of LLM calls, another set of TTFTs, another reasoning trace to wait through. The slowest model in the orchestra sets the tempo for the whole piece. This is why production teams increasingly fall back to deterministic pipelines for routine work: workflows are optimizable and run faster and cheaper than open-ended agents for tasks that do not need open-ended reasoning (Source: Gurusup, 2026). When you are paying latency at every node, the per-token speed of each model stops being a detail and becomes the dominant cost.

What does UX research say about latency thresholds?

It says the human limits are fixed and old. The Nielsen Norman Group’s three response-time limits, 0.1 seconds for “instantaneous,” 1 second to keep a user’s flow of thought uninterrupted, and 10 seconds before attention drifts, are rooted in human perceptual psychology and have held for decades (Source: Nielsen Norman Group, 2024).

These thresholds were written for clicks and page loads, but they govern AI interfaces just as strictly. A reasoning model that thinks for 60 seconds blows through the 10-second attention limit six times over; only a streaming interface with a fast TTFT keeps the user engaged, because partial output resets the perception clock. The business stakes are documented: Google’s analysis of mobile landing pages found 53% of visits are abandoned when a page takes longer than three seconds to load (Source: Google / Marketing Dive, 2017). Latency is a conversion and retention metric, and AI products inherit that physics.

How fast is “fast,” and how wide is the spread?

Wide enough that speed reorders a quality ranking. On the Artificial Analysis leaderboard, output speeds span from frontier reasoning configurations running under 60 tokens per second to lightweight models exceeding 500 tokens per second, roughly a 10x range across models that a capability board might list within a few points of each other (Source: Artificial Analysis, 2026).

Hardware widens the gap further. In Artificial Analysis’s system load test, an 8-way NVIDIA B200 configuration serving gpt-oss-120b reached a peak of 92,909 output tokens per second in aggregate and 403 tokens per second on a single query (Source: Artificial Analysis, 2026). Delivered speed is a function of model, provider, hardware, and region, all moving independently of the weights. A number that varies this much, and that you cannot read off a model card, is exactly what a live board like TokenDyno exists to measure. For the underlying metric, see LLM Tokens Per Second.

Should leaderboards include speed? Answering the objections

Yes, and the common objections do not hold. The strongest is that speed is “just an engineering detail” that will improve on its own. But Gartner’s own projection cuts against complacency: even as per-token inference cost is expected to fall over 90% by 2030, total token demand from reasoning and agents rises faster, so the wall-clock cost of a task is not guaranteed to shrink (Source: Gartner, 2026).

A second objection is that quality and speed cannot be combined into one rank without distorting both. That is correct, and it is the wrong goal. The answer is not a blended score but a two-axis view: report capability and speed side by side and let the reader weight them for their use case. A research assistant can trade speed for depth; a customer-support agent on a one-second budget cannot. A third objection is that speed is too volatile to rank. But volatility is the argument for measuring it continuously, not against. Prices and speeds already vary unequally across tasks and providers, which is precisely why they need tracking rather than assumption (Source: Epoch AI, 2025). For the full trade-off framing, see LLM Benchmark Leaderboard: Quality vs Speed.

How should you read a speed-aware leaderboard?

Read it as two columns that you weight by deadline. First, check whether the latency you care about is TTFT (for streaming chat), output speed (for long generations), or end-to-end time (for agents waiting on a complete object). Then read capability against that, not before it (Source: Artificial Analysis, 2026).

Three habits make the reading honest. Match the metric to the consumer: a human reading a stream cares about TTFT; an agent parsing JSON cares about end-to-end time. Check the conditions: server region, input length, and provider all move the number. And weight by your latency budget: a model two points stronger but 10x slower is wrong for a real-time agent and right for an overnight batch job. A ranking that surfaces both axes lets you make that call deliberately. For a maintained two-axis overview, see LLM Leaderboard 2026.

Frequently asked questions

Does inference speed matter for LLM quality?

Speed and quality are separate axes, but speed determines whether quality is usable. A more capable model that runs 10x slower can miss an agent’s timeout or a user’s attention window, delivering a worse outcome despite a higher benchmark score. For real-time and agentic use, delivered quality equals capability constrained by latency (Source: Artificial Analysis, 2026).

Why do reasoning models feel slow?

Because they generate a long internal chain of thought before any visible answer. That thinking phase can run from ten to over a hundred seconds, and maximizing reasoning effort has been measured to raise token usage by four to five times for only a few points of accuracy, so the wait is dominated by hidden tokens, not output rate (Source: Towards Data Science, 2025; Source: Artificial Analysis, 2026).

Should leaderboards include speed?

Yes, as a separate column rather than a blended score. Output speed spans roughly 10x across models that quality boards rank within a few points, and that speed depends on provider, hardware, and region in ways a model card cannot capture. Reporting capability and speed side by side lets readers weight them by deadline (Source: Artificial Analysis, 2026).

What is the difference between TTFT and tokens per second?

TTFT is the time from request to the first token; tokens per second is the generation rate after streaming begins. TTFT governs the first impression of a streaming interface, while tokens per second governs how fast a long answer completes. End-to-end latency combines both plus any reasoning time (Source: IBM, 2025; Source: Anyscale, 2025).

Why does latency compound in agentic workflows?

Because agents make many sequential LLM calls, and each call’s latency adds up. A four-agent, five-round debate is at least twenty calls, so a 10x difference in per-call speed becomes a 10x difference in total task time. Each reasoning step adds cost and end-to-end latency (Source: arXiv, 2026; Source: Gurusup, 2026).

Will faster hardware make speed irrelevant to rankings?

No. Even as per-token cost is projected to fall over 90% by 2030, agentic and reasoning workloads demand five to thirty times more tokens per task, so total wall-clock time is not guaranteed to shrink. Delivered speed still varies 10x across providers and configurations today (Source: Gartner, 2026; Source: Artificial Analysis, 2026).

Key takeaways

A model’s value is throughput times quality, not quality alone. Inference speed splits into TTFT, output speed, and end-to-end time, each tied to a different part of the experience, and all three vary with provider, hardware, and region in ways a capability score never captures. In agentic workflows, that variation compounds across every call, turning a 10x speed gap into a 10x difference in task completion time.

The recommendation is concrete: rank capability and speed as two co-equal axes, measure speed continuously because it is a property of the deployed system, and weight the two by your latency budget rather than assuming speed is free. A leaderboard that hides the speed column is not neutral; it is quietly optimizing for a use case that barely exists anymore.

Sources

← All posts