How do I benchmark vLLM vs SGLang on my own hardware?

Benchmarking vLLM against SGLang on your own hardware requires running both against the identical model checkpoint, quantization format, GPU, and prompt dataset, since results from public leaderboards rarely transfer to a different model size, context length, or traffic pattern. Start both servers with their OpenAI-compatible APIs and use a load generator such as vLLM's benchmark_serving.py or NVIDIA's genai-perf, which can replay a fixed dataset like ShareGPT at controlled request rates and report time to first token, inter-token latency, and total throughput as percentiles. Sweep several concurrency levels for each engine to build a throughput-versus-latency curve rather than comparing a single data point, and repeat the run a few times, since GPU thermal state and background processes introduce measurable noise. Pay attention to workload characteristics that favor one engine: SGLang's RadixAttention tends to show larger gains on prompts with heavy prefix reuse, while vLLM's broader quantization and hardware support can matter more for cost-constrained setups. Also record each engine's startup and warm-up time separately, since that affects autoscaling behavior even when it does not affect steady-state throughput. Nanobase AI runs exactly this kind of head-to-head benchmark as part of its serving engine selection engagements.

Setting up a fair comparison, concretely

Once you know which metrics matter for your product, running the actual vLLM versus SGLang comparison is a mechanical process, and most of the risk is in accidentally comparing two unfair configurations rather than in the tools themselves. Both engines expose an OpenAI-compatible server, which means the same load generator can drive both without modification, as long as the underlying setup is matched.

Fix everything except the engine itself before running a single request: identical model checkpoint, identical quantization format where both support it, identical GPU, and identical dataset, or the comparison measures configuration differences rather than engine differences.

Starting both servers with equivalent settings

# vLLM
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 --gpu-memory-utilization 0.90 --port 8000

# SGLang
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp 4 --mem-fraction-static 0.90 --port 8001

Match tensor-parallel degree and target memory utilization as closely as each engine's flags allow; small differences here will bias results before any traffic is sent.

Load generation and dataset choice

vLLM ships a benchmark_serving.py script capable of replaying a fixed dataset, commonly ShareGPT-derived conversation traces, at controlled request rates against any OpenAI-compatible endpoint, which means it works against SGLang's server too. NVIDIA's genai-perf tool serves the same purpose with additional protocol support and percentile reporting built in. Either tool can drive both engines since the target is the OpenAI-compatible API, not an engine-specific interface.

1. Choose a dataset that resembles your actual production prompt and output length distribution, not a generic benchmark set, since results are sensitive to this.
2. Run the load generator against vLLM first at a fixed concurrency level, recording TTFT, inter-token latency, and throughput percentiles.
3. Run the identical load generator configuration against SGLang at the same concurrency level.
4. Repeat both runs at several concurrency levels to build a throughput-latency curve for each engine, not a single data point comparison.
5. Repeat each run two or three times and check for variance, since GPU thermal state and background system activity introduce measurable noise between runs.
6. Record each server's cold-start and warm-up time separately, since this affects autoscaling behavior even though it doesn't reflect steady-state performance.

A results table structure that avoids misleading conclusions

Fill this from your own runs; do not substitute published numbers from either project's documentation, since those reflect their own test conditions, not yours.

Workload variants worth testing separately

Because SGLang's RadixAttention and vLLM's prefix caching behave differently on shared-context workloads, run at least two dataset variants: one with fully unique prompts and one with a shared system prompt or few-shot prefix repeated across requests. A single dataset that doesn't include prompt reuse will systematically understate any advantage RadixAttention offers on branching or shared-context traffic, which is exactly the scenario covered in vLLM vs SGLang for high throughput.

Frequently asked questions

Can the same load generator script test both engines without modification?

Yes, since both expose an OpenAI-compatible chat completions endpoint, the same script pointed at each server's respective port and base URL will work without engine-specific changes, which is the main practical benefit of API compatibility for benchmarking.

How many times should each benchmark run be repeated?

At least two or three repetitions per configuration is a reasonable minimum to catch run-to-run variance from GPU thermal throttling or background load; report the range or standard deviation alongside the central result rather than a single run's numbers.

Does quantization need to match exactly between the two engines?

As closely as each engine's supported formats allow; if one engine doesn't support the exact quantization format available on the other, note that as a limitation of the comparison rather than silently testing mismatched precisions.

Should the benchmark include a cold-start measurement?

Yes, as a separate metric from steady-state performance, since cold-start and model-load time affects autoscaling and deployment behavior even though it has no bearing on how the engine performs once warmed up and serving steady traffic.

How Nanobase AI helps

Nanobase AI runs exactly this kind of matched-configuration, multi-concurrency benchmark between vLLM and SGLang on a customer's own hardware and representative traffic before recommending a serving engine, rather than relying on published comparisons that reflect someone else's setup.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.