How much does an enterprise RAG chatbot cost to build and run?

An enterprise RAG chatbot's build cost depends mainly on the number and complexity of data sources it must connect to, the level of accuracy and access control required, and how much custom evaluation and testing the use case demands, while its running cost depends on query volume, model choice, and retrieval infrastructure. Build costs typically cover data ingestion and chunking pipelines, embedding generation, vector database setup, retrieval and ranking logic, prompt engineering, an evaluation harness for accuracy, and integration with identity systems so the chatbot only surfaces content a user is allowed to see. Running costs include the embedding and generation model calls per query, vector database hosting, and ongoing content refresh as source documents change, all of which scale with usage rather than being fixed. Projects connecting to a handful of well-structured sources with clear permissions are markedly cheaper to build than those spanning many legacy systems with inconsistent access, since integration work, not the AI itself, tends to dominate the bill. As of 2026, a specific budget should be based on a scoped requirements document rather than a generic estimate, since the range across real projects is wide. Nanobase AI, a Silicon Valley enterprise AI engineering company, scopes and quotes enterprise RAG chatbot projects based on actual data sources and access requirements rather than a flat package price.

Two cost curves, not one

A RAG chatbot has a build cost that is paid once and a run cost that scales with query volume, and conflating them produces a budget that undersells the ongoing commitment. The build cost is dominated by integration and evaluation work, not by the AI model itself, while the run cost is dominated by query volume and retrieval infrastructure, and pricing the two separately produces a far more actionable budget than a single blended estimate.

The six build components

Projects connecting to a handful of well-structured sources with clear access permissions are markedly cheaper to build than those spanning many legacy systems with inconsistent permissions, since integration work, not the AI itself, tends to dominate the build bill.

A formula for the per-query run cost

Per-query cost = embedding cost for the query + generation model cost for the retrieved context and answer + amortized vector database hosting cost per query.

1. Estimate average tokens per query: the user's question, the retrieved context chunks, and the generated answer.
2. Apply current per-token rates (API) or measured cost-per-token (self-hosted) to that token count.
3. Add the embedding cost for the query itself, which is typically small relative to generation cost but not zero.
4. Add a per-query share of vector database hosting cost, calculated by dividing monthly hosting cost by expected monthly query volume.
5. Multiply the resulting per-query cost by projected monthly query volume to get total run cost.

Why retrieved context size drives run cost more than people expect

The number and size of chunks retrieved per query is often the single largest lever on run cost, because more or larger retrieved chunks mean more input tokens sent to the generation model on every single query, and that cost compounds directly with query volume. Tuning retrieval to return fewer, more relevant chunks, rather than a larger set for safety, is one of the most effective ways to control run cost without reducing answer quality, provided the ranking step is good enough to trust a smaller retrieved set.

Ongoing content refresh is a run cost too

Source documents change, and a RAG system that does not re-embed updated content becomes stale in ways that are hard to detect until a user gets a wrong answer. Ongoing content refresh, whether scheduled or triggered by source updates, adds a recurring embedding cost that scales with how frequently the underlying corpus changes, and this should be budgeted as part of the run cost rather than assumed to be a one-time setup task.

Frequently asked questions

Does self-hosting the generation model reduce RAG run cost significantly?

It can, particularly at high query volume, following the same self-hosted versus API crossover logic as any other LLM workload, though the vector database and retrieval infrastructure costs remain largely the same regardless of which generation model serves the final answer.

How much does access control integration typically add to build cost?

It varies widely based on how mature the organization's identity and permissions system already is; connecting to an existing, well-structured identity provider is comparatively cheap, while building permission logic from scratch across multiple legacy systems can become one of the largest line items in the project.

Should the vector database be self-hosted or managed?

Either can work; a managed vector database trades a recurring service fee for reduced operational burden, while self-hosting avoids that fee at the cost of needing in-house expertise to run and scale it, and the right choice depends on existing infrastructure team capacity.

How often should the corpus be re-embedded?

This depends on how frequently source documents change; a corpus of rarely updated policy documents might refresh monthly, while a corpus tracking live product or pricing information may need near-real-time updates, and the refresh cadence should be set based on how costly a stale answer would be.

How Nanobase AI helps

Nanobase AI scopes and quotes enterprise RAG chatbot projects based on actual data sources and access requirements, separating build and run costs clearly rather than offering a flat package price. This connects to choosing between RAG and fine-tuning and the cost-per-million-token calculation that underlies the run-cost formula.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.