Cost per million tokens for a self-hosted LLM is calculated by dividing the fully loaded hourly cost of the serving infrastructure by its sustained token throughput, then scaling to a million-token basis. The infrastructure cost side should include amortized GPU hardware or cloud rental rate, electricity, networking, and a share of engineering and operations time, not just the bare GPU rental price, since ignoring overhead understates the real cost significantly. Throughput depends heavily on model size, quantization, batch size, and the serving engine, with tools like vLLM or TensorRT-LLM using continuous batching to substantially raise tokens generated per second per GPU compared with naive single-request serving. A practical approach measures actual output tokens per second under realistic concurrent load on the target hardware, converts that to tokens per hour, divides the all-in hourly cost by that figure, and multiplies by one million to get a comparable cost per million tokens. This number should be tracked separately for input and output tokens if the workload benefits from prompt caching, since cached input tokens cost far less to reprocess than freshly computed ones. Comparing the resulting figure against current API pricing gives a fair like-for-like decision basis. Nanobase AI builds this cost model using benchmarks on a customer's actual model and hardware rather than vendor marketing numbers.
Five inputs, one formula
Cost per million tokens is not a lookup, it is a calculation built from five inputs: infrastructure cost, throughput, batch efficiency, and the split between input and output tokens. Skipping straight to a per-token estimate without measuring actual throughput under realistic concurrent load is the most common reason self-hosted cost estimates turn out wrong once the system is in production. The formula itself is simple; the discipline is in measuring the inputs honestly rather than assuming vendor benchmark numbers apply to a specific deployment.
The variables that go into the formula
| Variable | What it means | Common mistake |
|---|---|---|
| All-in hourly infrastructure cost | Amortized hardware or cloud rental rate, plus electricity, networking, and a share of ops time | Using bare GPU rental price only, ignoring overhead |
| Sustained tokens per second | Measured under realistic concurrent load, not a single-request benchmark | Using a vendor's best-case single-stream number |
| Batch efficiency | How well the serving engine overlaps requests via continuous batching | Assuming naive one-request-at-a-time serving |
| Input vs output token split | Output tokens typically cost more to generate than input tokens cost to process | Treating all tokens as equivalent cost |
| Cache hit rate | Share of input tokens served from a prompt or KV cache instead of freshly computed | Ignoring caching entirely in the cost model |
The formula and a worked example
Cost per million tokens = (all-in hourly cost / sustained tokens per hour) x 1,000,000
Using illustrative values only, not real prices, to show the mechanics:
- Suppose the all-in hourly cost of a serving node, including amortized hardware, electricity, and a share of ops time, is an illustrative $H per hour.
- Suppose benchmarking under realistic concurrent load with a continuous-batching engine like vLLM or TensorRT-LLM measures an illustrative T output tokens per second, sustained.
- Convert to tokens per hour: T x 3,600.
- Cost per million tokens = ($H / (T x 3,600)) x 1,000,000.
- Repeat the measurement separately for input token processing, since prefill and decode have different throughput characteristics and should not share one blended rate if the application's input-to-output ratio is far from typical.
Why the throughput number is where estimates go wrong
A single-request latency benchmark measures something almost unrelated to production cost, because production serving relies on continuous batching to keep the GPU busy across many concurrent requests rather than processing one at a time. A serving engine that batches well can multiply effective throughput several times over compared with naive serving, which means the choice of inference engine, not just the GPU, has a direct and often underestimated effect on cost per million tokens. Benchmarking should always use the target production concurrency level, not a single isolated request.
Accounting for caching correctly
Prompt or KV cache reuse changes the calculation for applications with repeated system prompts, few-shot examples, or retrieved context that stays similar across requests. Cached input tokens cost far less to reprocess than tokens computed fresh, so a workload with a high cache hit rate should calculate a blended cost per million tokens that reflects the actual mix of cached and uncached input, not a single flat input rate. Ignoring this split tends to overstate the true cost for applications with long, mostly static system prompts.
Frequently asked questions
Should cost per million tokens be calculated separately for input and output?
Yes, output token generation is typically more computationally expensive than input token processing because decoding happens sequentially token by token while input processing can be parallelized, so a single blended rate can misrepresent the true cost for workloads with an unusual input-to-output ratio.
How does batch size affect the calculation?
Larger batch sizes generally improve GPU utilization and raise tokens-per-second throughput up to a point, which lowers cost per million tokens, but very large batches can also increase per-request latency, so the throughput measurement should reflect a batch size the application can actually tolerate.
Does quantization change this calculation?
Yes, lower-precision formats like FP8 typically increase tokens-per-second throughput on hardware built to accelerate them, which directly lowers cost per million tokens, though the throughput gain should be measured on the target hardware rather than assumed from published figures.
How often should this calculation be re-measured?
Whenever the model, serving engine version, hardware, or typical request pattern changes meaningfully, since throughput is sensitive to all of these, and a stale measurement can lead to under- or over-provisioning relative to actual cost per token.
How Nanobase AI helps
Nanobase AI builds this cost model using benchmarks on a customer's actual model, serving engine, and hardware rather than vendor marketing numbers, measuring throughput under realistic concurrent load before quoting a cost-per-token figure. This connects to choosing between vLLM, TensorRT-LLM, Ollama, and SGLang and sizing GPUs for large models. See /demo for a live benchmark example.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.