An array of open-source benchmarking instruments measuring an LLM engine, split between quality dials and a tokens-per-second speedometer, near-black on white with a single green data accent

Blog · By Anton Gulin · Updated September 11, 2026

LLM Benchmark Tooling: Run Your Own Tests

Public leaderboards tell you how a model performs on someone else’s prompts under someone else’s conditions. Benchmark tooling lets you run the test yourself, on your tasks and your hardware. The open-source tooling now splits cleanly into two families: capability evaluators like EleutherAI’s lm-evaluation-harness, which backs Hugging Face’s Open LLM Leaderboard (Source: EleutherAI, 2026), and speed harnesses like NVIDIA’s GenAI-Perf. This roundup covers nine, what each measures, and how to install and run it.

The distinction matters because the two families answer different questions. Capability tools ask “is the output correct?” Speed tools ask “how fast and how many at once?” Picking the wrong family is the most common mistake teams make when they first try to benchmark a model in-house. This guide keeps them separate and tells you when to reach for each.

What tools can you use to benchmark LLMs yourself?

The open-source toolkit divides into capability and speed. For capability, the established options are lm-evaluation-harness, HELM, OpenAI Evals, DeepEval, and Promptfoo. For speed and throughput, use the vLLM bench CLI, NVIDIA’s GenAI-Perf, LLMPerf, and Locust for custom load tests. All nine are free and run from the command line.

Capability tools score the quality of model outputs against reference answers or judge models. Speed tools generate load against a serving endpoint and report latency and throughput. A complete evaluation usually uses one from each family: a capability harness to confirm the model is good enough, then a speed harness to confirm it is fast enough under your concurrency. For the conceptual background on what these scores mean, see LLM Benchmark: A Complete Guide.

Tools at a glance

ToolMeasuresLicenseSource
lm-evaluation-harnessCapabilityMITgithub.com/EleutherAI/lm-evaluation-harness
HELMCapability (+ efficiency, bias)Apache-2.0github.com/stanford-crfm/helm
OpenAI EvalsCapabilityMITgithub.com/openai/evals
DeepEvalCapability (app-level)Apache-2.0github.com/confident-ai/deepeval
PromptfooCapability + securityMITgithub.com/promptfoo/promptfoo
vLLM benchSpeed / throughputApache-2.0docs.vllm.ai
GenAI-PerfSpeed / throughputBSD-3-Clausegithub.com/triton-inference-server/perf_analyzer
LLMPerfSpeed / throughputApache-2.0github.com/ray-project/llmperf
LocustLoad testing (general)MITgithub.com/locustio/locust

Which tools benchmark LLM capability?

Five mature open-source frameworks measure capability. lm-evaluation-harness covers over 60 academic benchmarks and backs the Open LLM Leaderboard (Source: EleutherAI, 2026). HELM adds efficiency and bias metrics. OpenAI Evals and DeepEval target custom and application-level tests, and Promptfoo pairs evaluation with security red-teaming. Each scores output quality, not speed.

The right choice depends on whether you are testing a model or a system. Academic harnesses score a raw model against fixed datasets. Application frameworks score a full pipeline, retrieval included, against your own criteria. The next four sections cover each tool’s scope, license, and run command so you can match the tool to your question.

lm-evaluation-harness (EleutherAI)

The Language Model Evaluation Harness is the de facto standard for academic capability testing. It supports over 60 standard benchmarks with hundreds of subtask variants and serves as the backend for Hugging Face’s Open LLM Leaderboard (Source: EleutherAI, 2026). It is MIT-licensed and works across Hugging Face, vLLM, and API-based model backends.

Install it with pip install lm_eval, optionally with a backend extra such as lm_eval[vllm] or lm_eval[api]. You then run a model against named tasks from the command line, for example evaluating on MMLU or HellaSwag in a single invocation. Reach for it when you want reproducible, leaderboard-comparable scores on established benchmarks rather than bespoke business logic (Source: EleutherAI, 2026).

HELM (Stanford CRFM)

HELM, the Holistic Evaluation of Language Models framework from Stanford’s Center for Research on Foundation Models, is built for breadth. Beyond accuracy it reports metrics like efficiency, bias, and toxicity, and includes standardized benchmarks such as MMLU-Pro, GPQA, IFEval, and WildBench (Source: Stanford CRFM, 2026). It is Apache-2.0 licensed and emphasizes reproducibility and transparency.

Install it with pip install crfm-helm. Use HELM when a single accuracy number is not enough and you need a multi-dimensional report covering fairness and efficiency alongside raw scores, for instance in research or governance contexts. Note that the project entered maintenance mode on June 1, 2026, so treat it as a stable reference rather than a fast-moving tool (Source: Stanford CRFM, 2026).

OpenAI Evals and DeepEval

OpenAI Evals is a framework for evaluating LLMs and LLM systems, shipping an open-source registry of benchmarks plus support for private, custom evals. It is MIT-licensed, requires Python 3.9 or newer, and installs with pip install evals (Source: OpenAI, 2026). It suits teams that want to encode their own task as a reusable eval and run it against any model.

DeepEval, from Confident AI, is Apache-2.0 and frames evaluation as Pytest-style unit testing for LLM applications. It scores correctness, relevancy, faithfulness, hallucination, bias, and toxicity, and ships built-in benchmarking on MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, and GSM8K (Source: Confident AI, 2026). Install with pip install -U deepeval. Choose it when you want eval cases living in your test suite and gating CI.

Promptfoo

Promptfoo is a CLI tool and library for evaluating and security-testing LLM applications. It combines side-by-side model comparison with built-in red-teaming and vulnerability scanning, making it as much a security tool as an eval framework (Source: promptfoo, 2026). It is MIT-licensed and the project notes it is now part of OpenAI.

Run it without installing via npx promptfoo@latest, or install globally with npm install -g promptfoo; it requires a recent Node.js runtime. Promptfoo fits teams that define test cases declaratively in YAML and want to compare prompts and models in CI while also scanning for prompt-injection and other vulnerabilities. It is the most product-oriented of the capability tools here.

Which tools benchmark LLM speed and throughput?

Four tools measure serving performance. The vLLM bench CLI ships with vLLM and benchmarks latency, online serving, and offline throughput. NVIDIA’s GenAI-Perf reports TTFT, inter-token latency, and throughput against any endpoint. LLMPerf load-tests OpenAI-compatible and other APIs, and Locust drives fully custom concurrency tests. All are command-line and free.

Speed tooling answers questions a capability harness cannot: time to first token, inter-token latency, and how throughput degrades as concurrency climbs. These metrics decide whether a model is viable for an interactive product. For the methodology behind measuring them correctly, see LLM Inference Speed Benchmark: Methodology. The next sections cover each speed tool’s scope and run command.

vLLM bench

If you serve models with vLLM, its built-in bench CLI is the path of least resistance. vLLM provides a vllm bench command with three subcommands: vllm bench latency for a single batch, vllm bench serve for online serving throughput, and vllm bench throughput for offline inference throughput (Source: vLLM, 2026). The project is Apache-2.0.

Install vLLM with pip install vllm; the bench commands are bundled, though some require extra benchmarking dependencies. The three-way split is the strength here: you can isolate single-request latency, online serving under concurrency, and raw offline throughput as separate measurements. Use it when vLLM is already your serving stack and you want numbers from the same engine you deploy (Source: vLLM, 2026).

GenAI-Perf (NVIDIA)

GenAI-Perf is NVIDIA’s generative-AI benchmarking CLI, living in the Triton perf_analyzer repository under a BSD-3-Clause license. It generates load against a model endpoint and reports a rich metric set: time to first token, time to second token, inter-token latency, request throughput, output token throughput (including per-user), request latency, sequence lengths, and GPU telemetry (Source: NVIDIA, 2026).

Install it with pip install genai-perf. It is the most metric-complete speed tool in this list and works well against any OpenAI-compatible or Triton-served endpoint. One caveat: NVIDIA’s docs note GenAI-Perf is being phased out in favor of a successor, AIPerf, so check the current docs before standardizing on it for new work (Source: NVIDIA, 2026).

LLMPerf and Locust

LLMPerf, from the Ray project, is an Apache-2.0 library for validating and benchmarking LLM API performance. It runs a load test and a correctness test, reporting TTFT, end-to-end latency, inter-token latency, per-request and overall output throughput, token counts, completed requests per minute, and error rate (Source: Ray project, 2026). It supports OpenAI-compatible, Anthropic, TogetherAI, Hugging Face, LiteLLM, Vertex AI, and SageMaker backends. Clone it and run python token_benchmark_ray.py. Note the repo is now archived, so treat it as stable but unmaintained.

Locust is a general MIT-licensed load-testing tool: you write user behavior in plain Python and swarm a target with concurrent users, and it scales across machines to simulate very high concurrency (Source: Locust, 2026). It is not LLM-specific, so you instrument token metrics yourself, but it is the most flexible option when your test needs custom request shaping or realistic traffic patterns. Install with pip install locust.

How do you choose between these benchmark tools?

Choose by the question you are answering. For leaderboard-comparable capability scores, use lm-evaluation-harness. For multi-dimensional capability reports, use HELM. For app-level and CI testing, use DeepEval or Promptfoo. For serving speed on vLLM, use vllm bench; for any endpoint, use GenAI-Perf or LLMPerf; for fully custom load, use Locust.

A sensible default pipeline is two tools. First, run a capability harness to confirm the model meets your quality bar on tasks resembling your workload. Then run a speed harness against the exact provider and hardware you intend to ship on, measuring TTFT and throughput at your real concurrency. Capability without speed ships a slow product; speed without capability ships a fast wrong answer. You need both numbers before committing.

How do I benchmark LLM speed myself?

Stand up the model behind an OpenAI-compatible endpoint, then point a speed harness at it. GenAI-Perf gives the most complete metric set out of the box with pip install genai-perf and a single run command. Measure at several concurrency levels, because throughput and inter-token latency change sharply under load, and record the hardware and context length so the run is reproducible (Source: NVIDIA, 2026).

The reason to test concurrency rather than a single request is that production traffic is concurrent. A model that streams 80 tokens per second to one user may halve that under fifty simultaneous requests on the same GPU. For the deeper methodology on throughput specifically, including batching effects, see LLM Throughput Benchmark. If you only need a published live cross-provider reference rather than a self-run test, a continuous tracker like TokenDyno provides one.

Frequently asked questions

What tools benchmark LLMs?

Open-source LLM benchmark tools split into two families. For capability there are lm-evaluation-harness, HELM, OpenAI Evals, DeepEval, and Promptfoo, which score output quality. For speed there are the vLLM bench CLI, NVIDIA’s GenAI-Perf, LLMPerf, and Locust, which measure latency and throughput. All nine are free and command-line driven.

How do I benchmark LLM speed myself?

Serve the model behind an OpenAI-compatible endpoint, then run a speed harness against it. GenAI-Perf (pip install genai-perf) reports TTFT, inter-token latency, and throughput; LLMPerf and the vLLM bench CLI are alternatives. Test at multiple concurrency levels and record the hardware, because throughput degrades under load (Source: NVIDIA, 2026).

What is the best open-source LLM eval tool?

There is no single best; it depends on your question. lm-evaluation-harness is the standard for leaderboard-comparable academic scores and backs Hugging Face’s Open LLM Leaderboard (Source: EleutherAI, 2026). For application-level testing in CI, DeepEval or Promptfoo fit better. For multi-dimensional reports including efficiency and bias, use HELM.

Are these LLM benchmark tools really free?

Yes. All nine are open-source under permissive licenses: MIT for lm-evaluation-harness, OpenAI Evals, Promptfoo, and Locust; Apache-2.0 for HELM, DeepEval, LLMPerf, and vLLM; and BSD-3-Clause for GenAI-Perf (Source: respective project repositories, 2026). You may incur costs for model API calls or GPU rental, but the tooling itself carries no license fee.

Can one tool measure both LLM capability and speed?

Not really; the two families are built for different questions. Capability tools score correctness against references or judges, while speed tools measure latency and throughput against a serving endpoint. HELM reports some efficiency metrics alongside accuracy, but for serious speed numbers you still need a dedicated harness like GenAI-Perf or the vLLM bench CLI (Source: Stanford CRFM, 2026).

Key takeaways

Benchmark tooling lets you replace borrowed leaderboard numbers with measurements on your own tasks and hardware. The open-source tooling splits into capability evaluators (lm-evaluation-harness, HELM, OpenAI Evals, DeepEval, Promptfoo) and speed harnesses (vLLM bench, GenAI-Perf, LLMPerf, Locust). All nine are free, command-line tools with verifiable licenses and active or stable repositories.

The practical recipe is two tools, one from each family: confirm quality with a capability harness, then confirm speed against your real provider and concurrency. Match the tool to the question, record your conditions, and test under load rather than with a single request. A few caveats for 2026: HELM is in maintenance mode, LLMPerf is archived, and GenAI-Perf is being succeeded by AIPerf, so check current docs before standardizing.

Sources

← All posts