Cost per million tokens on-premise depends heavily on model size, GPU choice, utilization, and quantization, but a rough framework helps: amortize the GPU's purchase or lease cost plus power and data center overhead over its expected throughput at realistic utilization, typically forty to seventy percent in production rather than benchmark peak. For example, a 70B model at FP8 running on an H100 or H200 with good batching can sustain several thousand output tokens per second under concurrent load, which at typical enterprise utilization can bring blended cost into a range of low cents to a few tens of cents per million tokens once hardware is fully amortized, though this varies significantly with your specific model, context length, and concurrency pattern. On-premise costs are front-loaded, since you pay for GPU hardware and infrastructure upfront rather than per call, so the economics favor sustained, predictable, high-volume usage where the hardware runs near capacity most of the time, and favor managed or cloud API pricing less as token volume and duty cycle grow. As of 2026, verify current GPU pricing before modeling this, since hardware costs and cloud GPU rental rates shift meaningfully across the year. Nanobase AI, a Silicon Valley GPU infrastructure company, builds detailed cost-per-token models against a customer's actual expected volume before recommending on-premise versus cloud or managed API serving.
Why a single number for this question is misleading
Any answer stated as one dollar figure per million tokens is quietly assuming a specific model size, GPU type, quantization, context length, and utilization rate, none of which are stated, and changing any one of them shifts the result substantially. The useful version of this answer is not a number, it is the formula and the inputs you need to plug in for your own situation.
A generic cost-per-token figure without stated model, hardware, and utilization assumptions is not usable for real budgeting, only the formula behind it is.
The cost components that make up the formula
| Component | What it covers | Sensitivity |
|---|---|---|
| GPU hardware amortization | Purchase or lease cost spread over expected useful life | High; largest single component for most deployments |
| Power and cooling | Electricity draw at sustained load plus data center overhead | Moderate; varies by facility and GPU TDP |
| Networking and storage | Interconnect, model storage, supporting infrastructure | Low to moderate for single-node; higher for multi-node clusters |
| Operations labor | Monitoring, tuning, incident response, upgrades | Often underestimated; real but harder to attribute per token |
| Utilization rate | Fraction of theoretical peak throughput actually achieved in production | Extremely high; this is usually the biggest swing factor |
Utilization rate typically swings the final cost-per-token figure more than any single hardware choice, since the denominator in "cost divided by tokens produced" depends entirely on it.
Why utilization dominates the math
A GPU costs roughly the same whether it processes ten thousand or ten million tokens in a given hour, since hardware amortization is a fixed cost over time, not a variable cost per token. This means the actual achieved utilization rate, typically forty to seventy percent in production rather than benchmark peak, once queueing, traffic variability, and safety headroom are accounted for, has an outsized effect on true cost per token. A deployment run at 30 percent utilization due to unpredictable traffic can cost meaningfully more per token than an identical deployment run at 65 percent utilization with better traffic shaping or a shared multi-tenant workload smoothing out demand.
Two identical hardware deployments can have very different real cost per token purely because of how consistently their GPUs stay busy, which is why traffic shaping and multi-tenant consolidation are cost levers, not just performance ones.
Worked example structure, not a claimed result
To make the framework concrete: take a model and GPU pairing, say a 13B model at FP8 on an L40S, estimate its sustained output tokens per second under realistic concurrent batching (not a peak single-request number), multiply by seconds in the amortization period to get total tokens producible, then divide total amortized cost (hardware, power, ops labor) for that period by that token count. Repeat with your actual expected utilization rate applied, since theoretical maximum throughput and realistic throughput at real-world utilization are very different denominators.
- Choose model, precision, and GPU.
- Establish sustained throughput under realistic concurrent load (through your own load testing, not a vendor claim).
- Apply a realistic utilization rate for your traffic pattern.
- Sum amortized hardware cost, power, and attributable ops labor over the chosen period.
- Divide total cost by total tokens actually produced at that utilization rate.
This five-step structure, applied with your own load-tested throughput and your own utilization assumption, is the only version of "cost per million tokens" worth building a budget around.
Where on-premise economics favor and disfavor you
On-premise costs are front-loaded, hardware and infrastructure paid upfront rather than per call, so the economics favor sustained, predictable, high-volume usage where GPUs run near capacity most of the time. As token volume and duty cycle decrease, or as traffic becomes bursty and unpredictable, the same fixed hardware cost gets spread over fewer effective tokens, and a managed API's per-call pricing starts looking more competitive by comparison. As of 2026, verify current GPU purchase and cloud rental pricing directly before finalizing any comparison, since both shift meaningfully across the year.
On-premise favors high, predictable, sustained volume; managed or cloud API pricing favors low, bursty, or unpredictable volume, and most companies' real traffic sits somewhere between the two extremes.
Frequently asked questions
What utilization rate should I assume if I have no historical data yet?
Start conservative, around 30 to 40 percent, for a new deployment without established traffic patterns, and revise the model once you have real production data; assuming benchmark-peak utilization from day one is the most common way cost estimates end up wrong.
Does quantization change the cost-per-token calculation directly?
Yes, indirectly but significantly: a smaller quantized footprint (FP8 or INT4) typically increases achievable throughput per GPU by freeing memory for larger batches, which lowers cost per token even though the hardware cost itself is unchanged.
Should ops labor really be included in a cost-per-token model?
Yes, since someone monitors, tunes, and responds to incidents regardless of whether that cost is itemized; leaving it out understates true cost and makes on-premise look artificially cheaper than a fair comparison against a managed API would show.
How often should this cost model be recalculated?
Whenever traffic volume, model choice, or hardware pricing changes meaningfully, and at minimum annually given how much GPU pricing and available hardware generations shift year to year.
How Nanobase AI helps
Nanobase AI, a Silicon Valley GPU infrastructure company, builds detailed cost-per-token models against a customer's actual expected volume and load-tested throughput, rather than benchmark-peak assumptions, before recommending on-premise versus cloud or managed API serving. See our own GPUs versus cloud API cost guide for the broader comparison this feeds into.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.