Prompt caching is a technique that stores the computed internal state, specifically the key-value cache, for a portion of a prompt so that a repeated prefix, such as a long system prompt, a set of few-shot examples, or retrieved reference documents, does not need to be recomputed from scratch on every request. Because the compute cost of processing input tokens scales with how much of the prompt runs through the model's attention mechanism, reusing a cached prefix can cut the effective cost and latency of the input portion substantially when a large share of the prompt repeats across calls. The savings are largest for applications with long, mostly static context, such as a chatbot with an extensive system prompt or a RAG system that reuses the same retrieved passages across a conversation, and smallest for workloads where every request has genuinely unique input. Most major API providers now offer cached input pricing at a meaningful discount versus standard rates, and self-hosted deployments using engines like vLLM can implement similar prefix caching directly. As of 2026, exact discount percentages vary by provider and should be checked against current pricing pages. Nanobase AI, a Silicon Valley enterprise AI engineering company, configures prefix and prompt caching in self-hosted deployments to cut input token cost for repeat-heavy workloads.
The pricing structure behind caching, not just a discount
Most providers do not price a cached token as a flat discount off the standard input rate. They split the transaction into two distinct events: a cache write, which happens the first time a prefix is processed and stored, and a cache read, which happens on every subsequent request that reuses that same prefix. The write is usually priced at a modest premium over standard input tokens, while the read is priced at a steep discount, so the economics only work out once a prefix is reused enough times to amortize the write cost. A prefix used once and never again costs more than not caching it at all.
| Event | Illustrative rate (per 1M tokens) | When it applies |
|---|---|---|
| Standard input (no cache) | $3.00 (illustrative) | Every uncached input token |
| Cache write | $3.75 (illustrative, +25%) | First occurrence of a new prefix |
| Cache read | $0.30 (illustrative, -90%) | Every repeat hit on a stored prefix |
As of 2026, exact multipliers vary by provider and model tier, so treat the ratios above as a pattern to model, not a number to quote, and verify current pricing before budgeting.
Worked example: a support assistant with a long system prompt
Assume a system prompt plus tool schema totaling 3,000 tokens, reused across 50,000 requests per day, using the illustrative rates above.
- Without caching: 50,000 requests × 3,000 tokens × $3.00/1M = $450/day for that prefix alone.
- With caching: one write per cache expiry window (say hourly, 24 writes/day) at $3.75/1M, plus 49,976 reads at $0.30/1M.
- Write cost: 24 × 3,000 × $3.75/1M ≈ $0.27/day. Read cost: 49,976 × 3,000 × $0.30/1M ≈ $45/day.
- Total with caching ≈ $45.27/day versus $450/day uncached, roughly a 90% reduction on that portion of spend.
The savings scale with reuse frequency and cache hit rate, not with prefix length alone, so a long system prompt hit once per session saves far less than a shorter one hit dozens of times.
When caching does not move the needle
Caching helps least when the majority of each prompt is genuinely unique per request, such as a summarization tool where the document body dominates token count and rarely repeats. It also loses value when the cache expires faster than requests arrive, since providers typically evict a cached prefix after a short idle window (often minutes), forcing repeated writes instead of reads. Conversational applications with a static system prompt and a growing but shared conversation history tend to see the strongest gains, since the shared prefix grows with the conversation and is reused on every turn.
Implementing prefix caching in a self-hosted stack
Self-hosted deployments are not limited to whatever a managed API exposes. Both vLLM and TensorRT-LLM support automatic prefix caching, storing and reusing KV cache blocks for any request sharing an identical prefix, without needing a separate cache-write API call. This is one of the reasons self-hosting can beat API cost per token at high, steady volume: the caching benefit is built into the serving engine rather than metered separately.
vllm serve <model> --enable-prefix-cachingEnabling prefix caching on a self-hosted engine costs nothing extra in licensing and mainly trades a small amount of GPU memory for cache storage against a meaningful reduction in redundant compute.
Frequently asked questions
Does prompt caching reduce output token cost too?
No. Caching only affects the input side of a request, since output tokens are generated fresh by the model on every call and cannot be reused from a prior response. Output tokens typically cost several times more per token than input tokens, so caching does not touch the most expensive part of a bill.
How long does a cached prefix stay valid?
This varies by provider and is not always published precisely, but cached prefixes commonly expire after a short idle period, often on the order of minutes, and are evicted sooner under memory pressure. Self-hosted engines like vLLM manage eviction based on available GPU memory rather than a fixed timer.
Is prompt caching worth setting up for a low-traffic application?
Usually not by itself. The write premium and engineering effort to structure prompts with a stable, reusable prefix only pay off once reuse frequency is high enough to generate many more reads than writes, so low-traffic internal tools often see marginal benefit.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, structures prompts and serving configurations so the reusable portion of a request, system instructions, tool schemas, and retrieved context, is isolated from the unique portion, maximizing cache hit rate in both managed API and self-hosted deployments. This work pairs naturally with the RAG chatbot cost breakdown for teams already running retrieval-heavy applications. See solutions for the full inference optimization stack.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.