Measuring LLM latency and throughput properly means testing under realistic concurrent load with a representative prompt and output length distribution, not a single sequential request, because inference engines behave very differently at a concurrency of one versus a concurrency of fifty. The key metrics are time to first token, inter-token latency or time per output token, end-to-end request latency, and throughput in tokens per second across the whole system, each reported as percentiles such as p50, p95, and p99 rather than just an average, since tail latency is what users actually notice. Tools like vLLM's own benchmark_serving script or NVIDIA's genai-perf can replay datasets such as ShareGPT conversations at controlled request rates and concurrency levels, producing exactly these percentile breakdowns. It is important to sweep multiple concurrency levels to find the throughput-latency curve and identify the point where adding more concurrent requests increases throughput without unacceptably degrading latency, since that saturation point determines real capacity per GPU. Testing should also match production conditions, including the same quantization, context length distribution, and hardware, since results do not transfer cleanly across GPU generations or precisions. Nanobase AI runs these load tests on customer hardware before committing to capacity or SLA numbers.
The decision that precedes any tool choice
Before selecting a benchmarking tool or running a single load test, the more important decision is which metrics actually reflect your product's user experience, because optimizing the wrong metric produces a deployment that benchmarks well and still disappoints users. This is a decision problem, not a tooling problem, and it should be settled before any commands get run.
Choose metrics that map to what a user actually experiences, then choose percentiles that reflect what your worst-served users experience, since an average across all requests hides exactly the failures that generate complaints.
The four metrics that matter, and what each one actually tells you
| Metric | What it captures | What it misses if used alone |
|---|---|---|
| Time to first token (TTFT) | How long a user waits before seeing any response | Says nothing about how fast the rest of the response streams |
| Inter-token latency (time per output token) | How smooth and fast the streamed response feels after it starts | Says nothing about the initial wait |
| End-to-end request latency | Total time from request to complete response | Blends queueing, prefill, and generation into one number, hiding which stage is the bottleneck |
| System throughput (tokens/sec across all requests) | How much total capacity the deployment delivers | Says nothing about any individual user's experience, which can degrade even as aggregate throughput rises |
Tracking only one of these produces a distorted picture: a system can have excellent throughput while individual users experience poor time to first token under load, and neither number alone reveals that tradeoff.
Why percentiles, not averages, decide whether a deployment is actually acceptable
An average latency number is dominated by the bulk of typical requests and can look perfectly healthy while a meaningful fraction of requests experience latency several times worse. Reporting p50, p95, and p99 separately reveals this: p50 shows typical experience, p95 shows what a notable fraction of users experience, and p99 shows the tail that generates the most visible complaints and support tickets. A deployment decision based on p50 alone routinely gets approved and then generates escalations once real traffic exposes the tail that the average never showed.
The throughput-latency curve, not a single data point
Every serving configuration has a curve, not a single number: as concurrent load increases, throughput rises until some resource saturates, after which adding more concurrent requests increases throughput only marginally while latency degrades sharply. Reporting a single concurrency level's results without showing where it sits on that curve makes it impossible to know whether the deployment has headroom for a traffic spike or is already near its saturation point. Finding that saturation point, and deciding how much headroom below it to operate at, is a business decision about acceptable risk, not purely a technical measurement.
A checklist for choosing what to measure before testing
- Decide which of TTFT, inter-token latency, end-to-end latency, and system throughput actually maps to a business or user-experience requirement for this specific product.
- Commit to reporting p50, p95, and p99 for every latency metric, not an average, before results come in and get cherry-picked.
- Plan to sweep multiple concurrency levels rather than testing one, so the throughput-latency curve, not a single point on it, informs the capacity decision.
- Match the test's quantization, context length distribution, and hardware to the actual production configuration, since results do not transfer across those changes.
- Decide the acceptable operating point on that curve, including deliberate headroom below saturation, before traffic actually reaches it.
Once these decisions are settled, the concrete tooling and commands to execute them, including a head-to-head engine comparison, are covered in benchmarking vLLM vs SGLang on your own hardware.
Frequently asked questions
Is average latency ever an acceptable metric to report?
It can supplement percentile reporting but should never replace it, since an average can look healthy while a meaningful share of requests experience unacceptable tail latency that the average hides completely.
How many concurrency levels should a proper benchmark sweep test?
Enough to identify the inflection point where latency starts degrading sharply as concurrency increases, typically a handful of levels spanning from light load to well past expected peak traffic, rather than a single level assumed to represent all conditions.
Does system throughput matter if per-request latency already looks good?
Yes, because throughput determines how much total capacity, and therefore how many GPUs, a given traffic volume actually requires; strong per-request latency at low concurrency says little about cost-efficient capacity planning at real scale.
Should benchmark results from public leaderboards be trusted for capacity planning?
Generally no, since public benchmarks rarely match your specific model, quantization, context length distribution, and hardware; they are useful for directional comparison between engines, not for sizing your own deployment's actual capacity.
How Nanobase AI helps
Nanobase AI, a Silicon Valley engineering firm, defines the right latency and throughput metrics for each customer's actual product before running any load test, then reports full percentile breakdowns across a concurrency sweep so capacity decisions are grounded in the real throughput-latency curve rather than a single favorable data point.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.