The most reliable way to estimate tokens per second before buying GPUs is to rent the candidate GPU type from a cloud provider and benchmark the actual model, quantization level and serving engine intended for production, since published figures rarely match a specific combination of model, precision, batch size and prompt characteristics exactly. Start from vendor-published or community benchmarks for a similar model and GPU pairing as a rough baseline, then validate with a short rented-GPU test using realistic prompts, typical context lengths and the expected concurrency pattern, since throughput at low concurrency looks very different from throughput once continuous batching is saturated. Tools built into vLLM and TensorRT-LLM report both time-to-first-token and steady-state tokens per second, which together say more about real user experience than a single aggregate throughput number would. Factor in that production traffic is rarely evenly distributed, so testing at the expected peak load rather than average load avoids underestimating hardware needs. This benchmark-first approach is more dependable than any generic online calculator, which cannot account for specific prompt patterns. Nanobase AI runs exactly this kind of rented-hardware benchmark for customers before they commit to a GPU purchase.
The rent-then-benchmark process, step by step
Published figures rarely match your specific combination of model, precision, prompt pattern and concurrency exactly, so a short rented-GPU test before purchase is the most reliable way to get a real number.
- Identify the candidate GPU type and the model, precision and serving engine you intend to run in production.
- Rent that exact GPU type from a cloud provider for a short window, hours rather than weeks, specifically for benchmarking.
- Deploy the real serving stack, not a simplified test harness, using the same quantization and configuration you plan for production.
- Generate load using prompts and output lengths that match your actual expected traffic, not a generic benchmark dataset, since throughput is highly sensitive to this.
- Record time-to-first-token and steady-state tokens per second at several concurrency levels, from light to your expected peak.
- Compare the measured numbers against vendor-published or community benchmarks for the same model and GPU pairing as a sanity check, not as a replacement for your own test.
Key takeaway: renting the exact target GPU for a few hours to run your real model and real prompts is cheap insurance against sizing a purchase off numbers that don't apply to your workload.
Tools that make this measurable
| Tool | What it measures | Fits well with |
|---|---|---|
| vLLM's built-in benchmark scripts | Throughput and latency at configurable concurrency | vLLM deployments |
| NVIDIA GenAI-Perf | Detailed latency percentiles, time-to-first-token, throughput | TensorRT-LLM, NIM, Triton |
| LLMPerf and similar open-source load testers | Realistic multi-client load simulation | Framework-agnostic HTTP endpoints |
| Custom load scripts using your production prompt logs | The most representative results, since it uses your actual traffic shape | Any serving engine |
Using a tool built for your serving engine typically produces the most directly comparable results, but a simple custom script replaying real production prompts, if you have historical data from a pilot or prior deployment, is often the single most valuable input, since it removes the guesswork about prompt and output length distribution entirely.
Key takeaway: purpose-built benchmarking tools give clean, comparable numbers, but replaying real production prompt patterns is what makes those numbers trustworthy for your specific deployment.
Testing at peak load, not average load
Production traffic is rarely evenly distributed across the day, so a benchmark run only at light or moderate concurrency will look fine while hiding the behavior that matters most: what happens at the busiest hour. Time-to-first-token and steady-state throughput both typically degrade non-linearly as concurrency approaches a GPU's ceiling, meaning the difference between testing at 70 percent of expected peak and testing at 100 percent of expected peak can be the difference between a passing and failing result. Building a margin into the test itself, benchmarking at 120 to 130 percent of the expected peak concurrency, gives a more honest picture of how much headroom the hardware actually has before real-world variability starts to bite.
Key takeaway: benchmark at and slightly above your expected peak concurrency, not your average, since that is where hardware limits actually show up.
What a rented benchmark cannot tell you
A short benchmark validates throughput and latency for the configuration tested, but it does not substitute for longer-running checks such as memory stability over many hours of continuous traffic, behavior under a sudden traffic spike well beyond the tested peak, or how the system recovers from a GPU or process failure mid-request. These operational characteristics matter for a production deployment but need a longer soak test or staged rollout to observe, rather than the few hours a purchase-decision benchmark typically covers. Treat the rented benchmark as answering "will this hardware handle the expected load," and plan a separate operational validation phase for "will this hardware stay healthy under it."
Key takeaway: a purchase-decision benchmark answers whether the hardware handles expected load, not whether it stays stable over days of real production traffic, which needs its own validation phase.
Frequently asked questions
How long does a proper GPU benchmark take to run?
A focused benchmark covering a few concurrency levels with realistic prompts typically takes a few hours of rented GPU time, including setup, which is a small cost relative to the risk of purchasing hardware sized on a mismatched published figure.
Is a generic online GPU calculator good enough instead of benchmarking?
Generic calculators are useful for a rough first estimate but cannot account for your specific prompt patterns, output lengths, quantization choice or serving engine configuration, all of which materially change real throughput, so they should not replace an actual benchmark before a purchase decision.
Should I benchmark with synthetic prompts or real production data?
Real production data, or a close approximation from a pilot deployment, produces far more trustworthy results, since synthetic prompt generators often do not match the length distribution and content characteristics of actual usage.
What if I don't have production traffic yet to base a benchmark on?
Use the closest available proxy, such as prompts collected during an internal pilot, sample data from the intended use case, or a competitor product's typical usage pattern if documented, and revisit the benchmark once real traffic data becomes available after initial rollout.
How Nanobase AI helps
Nanobase AI runs exactly this kind of rented-hardware benchmark for customers before they commit to a GPU purchase, using the real target model, quantization and, where available, real prompt patterns from a pilot deployment. We report time-to-first-token and steady-state throughput at and above expected peak concurrency, not just a single headline number. See our live demo or our guide to choosing a serving engine for the tooling behind this process.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.