Cost per token on your own GPUs equals the fully loaded annual cost of the servers (hardware amortized over 3 to 5 years, power, space, networking, software, staff) divided by the GPU-hours you actually use, divided again by the tokens one GPU-hour produces on your real traffic. Self-hosting wins when steady, predictable volume keeps the GPUs at 60 to 90 percent utilization; cloud APIs win when volume is low, spiky or unknown. Most enterprises end up hybrid: a self-hosted baseline plus API capacity for bursts and frontier models.
The full cost model for self-hosting
Most cost comparisons go wrong in the first step: they compare the GPU purchase price against an API price list. The right unit is the fully loaded annual cost of running the servers, which has seven line items.
- Server capex, amortized. The full server price (GPUs, CPUs, memory, NVMe, NICs), straight-line over 3 to 5 years. Three years matches common IT depreciation; five matches how long a GPU generation stays useful for inference. The choice alone swings the result by up to 40 percent.
- Power and cooling. An 8-GPU H100 SXM server draws about 10 kW at full load (700 W per GPU plus CPUs, memory, fans and NICs). Multiply average draw by the facility PUE (about 1.2 to 1.5) and by your electricity price.
- Colocation or data-center space. Rack space, cooling and remote hands, usually priced per kW of provisioned power. GPU racks need 10 to 40 kW cabinets.
- Networking. Top-of-rack switches, InfiniBand or 400 GbE for multi-node setups, plus a share of transit and firewalls.
- Software and licenses. vLLM, TensorRT-LLM and NVIDIA NIM can be free to run, but enterprise Linux, monitoring, secrets management and optional NVIDIA AI Enterprise support are real line items.
- Operations staff. Patching, monitoring, model upgrades, re-benchmarking and on-call: a fraction of an FTE per server that shrinks as the fleet grows.
- Hardware support and spares. Typically a few percent of capex per year.
The eighth factor is not a cost but a divisor: utilization. A GPU you paid for costs the same whether it serves 2,000 tokens per second or sits idle at 3 a.m. The single most important number in the model is the share of purchased GPU-hours that actually serve traffic.
From annual cost to cost per GPU-hour and per million tokens
Once the annual cost is known, the conversion is two divisions: by the GPU-hours you will really use, then by the tokens one GPU-hour produces on your workload.
cost_per_gpu_hour = total_annual_cost / (gpu_count x 8,760 x utilization)
tokens_per_gpu_hour = measured_tokens_per_sec_per_gpu x 3,600
cost_per_million_tok = cost_per_gpu_hour x 1,000,000 / tokens_per_gpu_hourThe throughput term must be measured, not copied from a marketing slide. It depends on model size, quantization (FP8 or INT4 weights raise tokens per second substantially versus FP16), prompt and completion lengths, concurrency and the latency you accept. The same H100 can differ by 5x to 10x across those choices.
A defensible measurement procedure:
- Fix the model, quantization and serving engine you will run in production (see our serving engine comparison).
- Sample real traffic: input lengths, output lengths and requests per second by hour of day. Synthetic 128-token prompts overstate throughput.
- Load-test with the vLLM benchmark tools at increasing concurrency, replaying that distribution.
- Record aggregate tokens per second at the highest concurrency where p95 latency still meets your SLO.
- Divide by the GPUs in the test and apply a 15 to 25 percent headroom factor for peaks and model reloads.
If traffic is prompt-heavy, measure prefill and decode separately, because input tokens are processed in parallel and cost far less than output tokens generated one at a time. Measure throughput on your own traffic mix and treat every vendor benchmark as an upper bound.
Worked example with hypothetical numbers
The figures below are illustrative placeholders, not quotes. Real values vary by region, vendor and contract, so verify current pricing and replace every line with your own numbers.
| Line item (per year) | Assumption (hypothetical) | Annual cost |
|---|---|---|
| Server capex, amortized | 8-GPU server at $250,000, 4-year straight line | $62,500 |
| Electricity | 9 kW average draw x PUE 1.3 x 8,760 h x $0.12/kWh | ~$12,300 |
| Colocation space and cooling | $1,200 per month for one high-density cabinet share | $14,400 |
| Networking | Switch share, transit, firewall | $8,000 |
| Software and licenses | OS subscriptions, monitoring, optional enterprise support | $10,000 |
| Hardware support and spares | 5 percent of capex | $12,500 |
| Operations staff | 0.25 FTE at $200,000 fully loaded | $50,000 |
| Total | ~$169,700 |
Eight GPUs provide 70,080 GPU-hours per year. At 90 percent utilization the fully loaded cost is about $2.69 per GPU-hour; at 60 percent it is $4.04; at 30 percent it is $8.07. Nothing about the hardware changed, only how much of it you used.
Now apply a measured throughput. The table shows cost per million tokens (blended input and output) for three hypothetical per-GPU throughput levels against three utilization levels.
| Measured throughput per GPU (hypothetical) | 30% utilization | 60% utilization | 90% utilization |
|---|---|---|---|
| 150 tokens/s (large model, long context, low concurrency) | $14.95 | $7.47 | $4.98 |
| 300 tokens/s (mid-size model, FP8, moderate concurrency) | $7.47 | $3.74 | $2.49 |
| 600 tokens/s (small model or aggressive batching) | $3.74 | $1.87 | $1.25 |
The spread is 12x between the worst and best cell on identical hardware. Utilization and measured throughput each move cost per token by 3x or more, so the engineering decisions made after the purchase matter as much as the purchase itself.
How to compare against cloud API pricing
API prices are quoted per million tokens, but rarely as one number, so you need a blended rate that reflects your traffic. Verify current pricing with each vendor first; as of 2026 list prices change several times a year and differ by model tier.
Three pricing dimensions matter most:
- Input versus output tokens. Output tokens are priced several times higher than input at most vendors, because decode is the expensive phase. Summarization (long input, short output) and code generation (short input, long output) can differ by 3x in blended price on the same model.
- Batch discounts. Most vendors discount asynchronous batch jobs, commonly to around half the standard rate, in exchange for a completion window of hours.
- Prompt caching. Repeated prefixes (system prompts, RAG context, few-shot examples) are served from cache at a fraction of the input price, sometimes with a write fee. A high hit rate changes the math for agent and RAG workloads.
The blended API price per million tokens is:
p_blended = share_in x (hit_rate x p_cached + (1 - hit_rate) x p_in) + share_out x p_outHere share_in and share_out are the fractions of input and output tokens in your traffic; apply the batch discount to the asynchronous share. Compare the result against your self-hosted cost at the utilization you realistically expect, not at 100 percent.
| Dimension | Own GPUs | Cloud API |
|---|---|---|
| Cost structure | Fixed annual cost; marginal token nearly free until capacity is reached | Pure variable cost per token; no fixed base |
| Batch discount | Free: run offline jobs at night to raise utilization | Vendor discount for asynchronous jobs; verify current pricing |
| Caching | Prefix caching in vLLM or TensorRT-LLM at no extra charge | Discounted cached-input rate, sometimes with a write fee |
| Capacity ceiling | Hard limit of what you bought; bursts overflow or queue | Rate limits and quotas, negotiable at volume |
Compare your realistic self-hosted cost per million tokens at expected utilization against a blended API rate that reflects your input-output ratio, cache hit rate and batch share.
Break-even logic and hidden costs on both sides
Because self-hosting is a fixed cost and APIs are a variable cost, break-even is a monthly token volume: monthly fixed cost divided by the blended API price. In the worked example the server costs about $14,100 per month. Against a hypothetical blended API rate of $3 per million tokens, break-even is roughly 4.7 billion tokens per month, about 75 percent of the server's capacity at 300 tokens per second per GPU. Against a hypothetical $5 rate it drops to about 2.8 billion tokens, or 45 percent of capacity.
Steady, high, predictable volume therefore favors your own GPUs, because you can fill the capacity you bought. Spiky, low or unknown volume favors APIs, because idle hours cost nothing and you can change models without stranding hardware. Volume that is high on average but peaks at 5x the mean is the hard case that leads to hybrid designs.
Hidden costs on the self-hosted side:
- Procurement lead time and the engineering months before the first production token.
- GPU failures and firmware issues; a spare GPU or next-business-day support contract is not optional.
- Model churn: a new open-weight model needing 2x the memory can strand a server sized for its predecessor (see how many GPUs each model needs).
- Idle power: GPUs draw a meaningful share of peak power even when idle, so low utilization does not cut the electricity bill proportionally.
Hidden costs on the API side:
- Prompt bloat: system prompts, tool schemas and RAG chunks are re-sent on every call and billed as input every time.
- Price changes and model deprecations that force re-testing on the vendor's schedule.
- Rate limits at the exact moment your traffic peaks.
- Data protection: DPAs, residency guarantees and audit rights cost legal and compliance time.
Break-even is a volume, not a date: find the monthly tokens at which fixed self-hosting cost equals variable API cost, then ask honestly whether you will sustain it.
A hybrid strategy that usually wins
The cost model rarely produces a pure answer, and the most robust architecture as of 2026 uses both. Size a self-hosted cluster for the steady baseline (roughly the p50 to p70 of hourly load), run it at high utilization, and route everything above that to an API. Because vLLM, TensorRT-LLM and NVIDIA NIM expose an OpenAI-compatible endpoint, routing is a gateway configuration rather than an application rewrite.
Routing rules that keep the model honest:
- By data sensitivity first: regulated or confidential prompts stay on your GPUs regardless of cost.
- By model: a mid-size open-weight model on your GPUs handles the bulk; the API serves the small share that needs a frontier model.
- By urgency: interactive traffic gets the self-hosted baseline; asynchronous jobs go to the API batch tier when the cluster is full and to your own GPUs at night when it is not.
- By capacity: when queue depth exceeds a threshold, overflow to the API and log the cost so the next sizing round has data.
A useful intermediate step is renting whole cloud GPUs by the hour or year: it removes procurement lead time and produces real utilization data before you commit capex. Many teams start there, then move the baseline to owned hardware following an on-premise deployment plan. Nanobase AI typically recommends this sequence when traffic history is shorter than six months.
Own the baseline, rent the peaks, and make the routing decision in a gateway so the split can change without touching applications.
Frequently asked questions
What is a realistic GPU utilization rate for self-hosted LLM inference?
Clusters that serve only interactive daytime traffic typically land at 30 to 50 percent utilization because nights and weekends are idle. Teams that add batch workloads (document processing, evaluations, embeddings) in the off-hours reach 70 to 90 percent. Plan at 60 percent unless you have measured data; anything above 85 percent sustained is excellent.
Should I amortize GPU servers over 3 or 5 years?
Use 3 years for the conservative case and 5 years for the optimistic case, and show both. Three years aligns with common IT depreciation schedules and with the pace of new GPU generations. Five years reflects that an H100 still serves inference well after newer parts ship, at a higher cost per token than the newest hardware. If your finance team mandates a schedule, use it.
How do I measure tokens per second for the formula?
Load-test the exact model, quantization and serving engine you will run, replaying your real prompt and completion length distribution. Increase concurrency until p95 latency breaks your SLO, and take the aggregate tokens per second just below that point. Divide by GPU count. Re-measure after every model or engine upgrade, because throughput changes more than most teams expect.
Why are output tokens more expensive than input tokens?
Input tokens are processed in one parallel prefill pass that uses GPU compute efficiently. Output tokens are generated sequentially, one forward pass per token, and each pass is bound by memory bandwidth rather than compute. That is why H200 and B200, with higher memory bandwidth, improve decode throughput. API vendors price output higher for the same reason, so your input-to-output ratio drives the blended cost.
Does prompt caching change the self-hosting decision?
It can. If most input tokens are repeated prefixes such as long system prompts or shared RAG context, API caching cuts the effective input price sharply, which lowers the blended API rate and pushes the break-even volume up. Self-hosted engines also support prefix caching, which raises your measured throughput. Model both effects with your real cache hit rate.
At what monthly token volume does self-hosting start to pay off?
There is no universal number because it depends on your hardware cost, utilization and the API rate you would otherwise pay. The method is fixed: divide your monthly fully loaded server cost by the blended API price per million tokens. In this article's hypothetical example that is a few billion tokens per month for one 8-GPU server. Below that volume, and for spiky traffic, APIs are cheaper.
How Nanobase AI can help
Nanobase AI builds the cost model with you and then delivers the infrastructure it recommends. We size H100, H200, B200 or RTX PRO clusters against your measured traffic, benchmark throughput with vLLM, TensorRT-LLM or NVIDIA NIM on your candidate models, and produce a cost-per-token comparison against current API pricing that your finance team can audit. If the answer is hybrid, we deploy the self-hosted baseline with Kubernetes GPU Operator, monitoring and an OpenAI-compatible gateway that routes overflow to AWS, Azure or Google Cloud.
As a Silicon Valley enterprise AI engineering company and NVIDIA Inception Program member, we cover hardware sizing, serving optimization and integration in one project. Explore our solutions or book a live demo.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.