Grid of differently sized accelerator chips linked by data pathways, with a tall green tokens-per-second bar rising from the highest-bandwidth chip on a white field

Blog · By Anton Gulin · Updated September 4, 2026

GPU Benchmark for LLMs: How Hardware Shapes Throughput

When you run a GPU benchmark for LLMs, the number that moves most is not FLOPs. It is memory bandwidth. An NVIDIA H200 carries the same compute core as an H100 but adds 43% more bandwidth, and that single change delivers roughly 43% more decode throughput at the same batch size (Source: NVIDIA, 2024). Hardware shapes tokens/sec, but not in the way spec sheets suggest.

This piece explains why. It walks through the four hardware levers that decide LLM inference performance, compares the major accelerator classes with verified specs, shows why token generation is bounded by memory bandwidth rather than raw math, and lays out how to benchmark a GPU for serving. Every figure below comes from a named 2025-2026 source with a link.

Which hardware specs actually determine LLM throughput?

Four specs dominate: memory bandwidth (how fast weights move from HBM), VRAM capacity (whether the model and KV cache fit), FLOPs (raw compute for prefill), and interconnect (how fast linked GPUs exchange data). For single-stream token generation, bandwidth is the binding constraint; FLOPs sit mostly idle (Source: arXiv, 2024).

The reason is structural. During autoregressive decoding, a model generates one token per forward pass, and each pass must read the entire weight matrix from memory. At batch size 1, transformer decode runs at roughly 1-2 FLOPs per byte of data moved, which is 200-600x below the compute roofline of a modern GPU (Source: arXiv, 2024). The chip spends its time waiting on memory, not calculating.

Why is memory bandwidth the binding constraint in decode?

Because every generated token requires re-reading the model’s weights. A 70B-parameter model at FP16 moves roughly 140 GB of data per token step. On an H100 SXM5 with 3.35 TB/s of bandwidth, that transfer alone takes about 42 ms per token at the theoretical maximum (Source: arXiv, 2024).

That arithmetic is why the H100-to-H200 jump matters despite identical compute. NVIDIA upgraded the H200’s memory subsystem from 80 GB of HBM3 at 3.35 TB/s to 141 GB of HBM3e at 4.8 TB/s, leaving the GH100 compute die unchanged (Source: NVIDIA, 2024). For memory-bound decode, more bandwidth is the upgrade. The same logic explains why specialized chips that hold weights in on-chip SRAM post the fastest per-user numbers, which we will reach below.

When does VRAM capacity become the bottleneck instead?

VRAM capacity becomes binding when the model weights plus KV cache do not fit on the device. A 405B-parameter model at FP8 needs roughly 405 GB just for weights, which forces multi-GPU sharding on any single accelerator below that capacity. Capacity sets what you can run; bandwidth sets how fast.

This is where AMD’s strategy targets NVIDIA directly. The Instinct MI300X ships 192 GB of HBM3 and the MI325X carries 256 GB of HBM3e, against 141 GB on the H200 (Source: AMD, 2024). Larger VRAM lets a single GPU or smaller cluster hold bigger models and longer context, reducing the cross-GPU traffic that interconnect has to absorb. Capacity and bandwidth are separate axes, and a benchmark has to report both.

How do the major accelerator classes compare on paper?

The headline gap across current accelerators is bandwidth, which spans from 3.35 TB/s on H100 to 8 TB/s on Blackwell and AMD CDNA 4, then jumps three orders of magnitude on wafer-scale SRAM designs. Capacity ranges from 80 GB on H100 to 288 GB on the MI355X (Source: AMD, 2025; NVIDIA, 2025).

These are theoretical peaks, not delivered tokens/sec, and the two correlate loosely. A chip with double the bandwidth does not automatically double real throughput, because software maturity, batching, and precision support all intervene. Still, the spec table below is the right starting point because it bounds what each class can do. Every row carries its own source.

Verified accelerator specifications (2025-2026)

AcceleratorMemory / bandwidthNotable forSource
NVIDIA H100 SXM80 GB HBM3 / 3.35 TB/sHopper baseline, NVLink 4 (0.9 TB/s)NVIDIA, 2022
NVIDIA H200 SXM141 GB HBM3e / 4.8 TB/sMemory-upgraded Hopper, +43% bandwidthNVIDIA, 2024
NVIDIA B200192 GB HBM3e / ~8 TB/sBlackwell flagship, FP4, NVLink 5 (1.8 TB/s)NVIDIA, 2025
NVIDIA GB200 NVL7272x B200, 8 TB/s/GPURack-scale MoE inference, 72-GPU NVLink domainNVIDIA, 2025
AMD Instinct MI300X192 GB HBM3 / 5.3 TB/sLarge VRAM on CDNA 3AMD, 2023
AMD Instinct MI325X256 GB HBM3e / 6 TB/sLargest in-class single-GPU HBMAMD, 2024
AMD Instinct MI355X288 GB HBM3e / 8 TB/sCDNA 4, native FP4/FP6AMD, 2025
Groq LPU230 MB SRAM / 80 TB/s on-chipDeterministic low latency; many chips per modelGroq, 2024
Cerebras WSE-344 GB SRAM / 21 PB/s on-chipWafer-scale; entire model in SRAMCerebras, 2024
Google TPU v7 (Ironwood)192 GB HBM3e / 7.4 TB/s per chipInference TPU; 9,216-chip podsGoogle, 2025

Why do H100, H200, B200, and GB200 produce such different numbers?

Because NVIDIA widened bandwidth and interconnect at each step while compute scaled too. In MLPerf Inference v5.0, eight B200 GPUs served Llama 2 70B at 98,443 tokens/sec in the server scenario, against 33,072 tokens/sec for eight H200 GPUs on the same task, roughly a 3x generational jump (Source: NVIDIA, 2025).

The B200 pairs ~8 TB/s of HBM3e with fifth-generation NVLink at 1.8 TB/s per GPU, double the H100’s interconnect (Source: NVIDIA, 2025). GB200 NVL72 then links 72 B200 GPUs into one NVLink domain, and NVIDIA reports it delivering roughly 15x the performance of an H200 system on a mixture-of-experts model like DeepSeek-R1 in InferenceMAX (Source: NVIDIA / SemiAnalysis, 2025). Aggregate throughput scales with bandwidth and linked-GPU count, not headline FLOPs alone.

Where do AMD MI300X and MI325X fit?

AMD competes on capacity per GPU. The MI300X offers 192 GB of HBM3 at 5.3 TB/s and the MI325X raises that to 256 GB of HBM3e at 6 TB/s, both exceeding the H200’s 141 GB at 4.8 TB/s (Source: AMD, 2024). The 2025 MI355X pushes to 288 GB of HBM3e at 8 TB/s on the CDNA 4 architecture with native FP4 and FP6 support (Source: AMD, 2025).

More VRAM means a single MI325X can hold a larger model or longer context before sharding, which cuts interconnect pressure. The practical caveat is software: NVIDIA’s inference stack is more mature, so delivered tokens/sec depends heavily on framework support for a given model. The spec advantage is real; converting it to throughput is a benchmarking question, not a datasheet one.

How do Groq, Cerebras, and TPU change the equation?

They move weights off HBM entirely or feed compute with far more memory bandwidth. Cerebras WSE-3 holds 44 GB in on-chip SRAM with roughly 21 PB/s of bandwidth, around 2,600x an H100’s HBM, which lets it stream tokens without HBM round-trips (Source: Cerebras, 2024). Groq’s LPU uses 230 MB of SRAM at about 80 TB/s per chip (Source: Groq, 2024).

That architecture explains the per-user speed leaderboard. On Llama 4 Maverick, Artificial Analysis measured Cerebras at 2,522 tokens/sec per user and Groq’s LPU at 549, against 1,038 for NVIDIA Blackwell (Source: Cerebras / Artificial Analysis, 2025). The trade-off is capacity: because each Groq chip holds only 230 MB, hundreds of chips are needed for a 70B model, shifting the bottleneck to the cluster fabric. Google’s TPU v7 Ironwood takes a third path, pairing 192 GB of HBM3e at 7.4 TB/s per chip into 9,216-chip pods (Source: Google, 2025). For a provider-level view of how these land in practice, see our LLM speed comparison.

Why doesn’t more FLOPs simply mean more tokens per second?

Because decode rarely uses the FLOPs. At batch size 1, a transformer’s decode step has an arithmetic intensity of roughly 1-2 FLOPs per byte, while a modern GPU needs around 200+ FLOPs per byte to keep its compute units busy. Below that threshold, the chip is memory-bandwidth bound and extra FLOPs sit idle (Source: arXiv, 2024).

This is the roofline model in action: plot performance against arithmetic intensity, and low-intensity workloads cap out on the memory ceiling, never reaching the compute ceiling. FLOPs matter most for prefill, where a long prompt is processed in parallel at high intensity. For the token-by-token generation users actually watch stream, bandwidth is destiny, which is why bandwidth-upgraded chips like the H200 win on decode despite unchanged compute (Source: NVIDIA, 2024).

How does batching shift the bottleneck from memory to compute?

Batching raises arithmetic intensity by amortizing each weight read across many requests. A single weight matrix loaded from HBM serves the whole batch, so at batch size 64 a 70B model’s intensity climbs toward 50-100 FLOPs per byte, and at batch 512+ some workloads approach compute-bound territory on an H100 (Source: arXiv, 2024).

That is why the same GPU posts two very different numbers. Packing more concurrent requests pushes aggregate tokens/sec up while each user’s per-token latency rises, a throughput-versus-interactivity trade-off NVIDIA models as a Pareto frontier (Source: NVIDIA, 2025). A benchmark that does not state its batch size and latency target is reporting an arbitrary point on that curve. We cover the aggregate side in depth in our LLM throughput benchmark.

How do you benchmark a GPU for LLM serving?

You sweep concurrency against a fixed model and report tokens/sec at each latency target, rather than quoting one peak number. The two standard tools are vLLM’s benchmark_serving.py, which streams synthetic requests at a live endpoint and measures throughput and latency, and NVIDIA’s GenAI-Perf, which runs concurrency sweeps over values like 64, 96, 128, 192, and 256 (Source: NVIDIA, 2025).

The goal is to find where rising concurrency stops improving aggregate throughput, the point at which memory bandwidth saturates. Below that point you are latency-bound; above it, adding requests only raises per-user latency without lifting total tokens/sec. A Prometheus and Grafana stack exposes KV cache occupancy and GPU utilization so you can see whether the device is memory- or compute-bound at each step.

What should a GPU LLM benchmark always report?

It should fix and disclose five variables: the exact model, the precision (FP4, FP8, or BF16), input and output token lengths, the batch size or concurrency level, and the latency target held. Change any one and tokens/sec moves, often by multiples.

Standardized suites do this for you. MLPerf Inference, run by MLCommons, fixes the model and a latency constraint, then reports server and offline throughput, which is why its numbers are comparable across vendors (Source: MLCommons, 2025). NVIDIA’s open-source InferenceMAX re-runs nightly against improving software, so even audited figures drift week to week (Source: NVIDIA / SemiAnalysis, 2025). Treat any single tokens/sec figure as a dated snapshot tied to a specific configuration, not a hardware constant. For the current top of the field, see our guide to the fastest LLM inference in 2026.

Frequently asked questions

What is the best GPU for LLM inference?

There is no single best; it depends on model size and workload. For maximum per-GPU throughput, NVIDIA’s B200 leads, serving Llama 2 70B at about 3x an H200 in MLPerf v5.0 (Source: NVIDIA, 2025). For fitting large models on one device, AMD’s 256-288 GB MI325X and MI355X lead. For lowest per-user latency, Cerebras and Groq lead.

Why does memory bandwidth matter for LLMs?

Because token generation re-reads the entire model from memory for every token produced. A 70B model at FP16 moves roughly 140 GB per token step, so on a 3.35 TB/s H100 the memory transfer alone takes about 42 ms (Source: arXiv, 2024). Decode is memory-bandwidth bound, so faster memory directly raises tokens/sec.

How do you benchmark a GPU for LLMs?

Pick a fixed model and precision, then sweep concurrency using vLLM’s benchmark_serving.py or NVIDIA GenAI-Perf, recording tokens/sec and latency at each level (Source: NVIDIA, 2025). Find the concurrency where throughput plateaus, which marks memory-bandwidth saturation. Always disclose model, precision, sequence lengths, batch size, and latency target.

Does more VRAM make an LLM faster?

Not directly. VRAM capacity determines whether a model and its KV cache fit on the device; bandwidth determines decode speed. A larger VRAM GPU like the MI325X (256 GB) can hold bigger models without sharding (Source: AMD, 2024), which reduces interconnect overhead, but two GPUs with equal bandwidth and different capacity generate single-stream tokens at similar speeds.

Why do Groq and Cerebras beat GPUs on per-user speed?

They keep model weights in on-chip SRAM instead of external HBM. Cerebras WSE-3 offers roughly 21 PB/s of SRAM bandwidth and Groq’s LPU about 80 TB/s, versus 3.35-8 TB/s of HBM on GPUs (Source: Cerebras, 2024; Groq, 2024). That eliminates the HBM round-trips that pace memory-bound decode, yielding the highest measured tokens/sec per user.

How often do GPU LLM benchmark numbers change?

Frequently, often within days. NVIDIA’s InferenceMAX re-runs nightly against evolving inference software, and providers re-optimize endpoints continuously (Source: NVIDIA / SemiAnalysis, 2025). A figure measured this week on one framework version may not hold next week, so always confirm against a current, dated source rather than a static datasheet.

Sources

← All posts