RAG vs fine-tuning: a decision guide for enterprise LLMs

Use RAG when answers must come from documents that change, must cite their sources, or must respect per-user access rights; that describes most enterprise knowledge work. Use fine-tuning when you need consistent style, output format or domain-specific behavior that prompting cannot deliver reliably, and that behavior is stable enough to justify a training run. Most mature deployments end up with both: RAG supplies the facts, and a small LoRA adapter shapes how the model responds.

What RAG and fine-tuning actually do

Retrieval-augmented generation (RAG)

RAG leaves the model untouched and changes what it sees at inference time. The question is embedded, matched against an index of your documents, and the best-matching passages are inserted into the prompt. The model answers from that context and can quote the passages it used. Knowledge lives in the index, so an update means re-indexing a document, not retraining.

Fine-tuning

Fine-tuning changes the model's weights by continuing training on your own examples. Full fine-tuning updates every parameter; parameter-efficient methods such as LoRA train small adapter matrices and keep the base weights frozen. The result follows your format, uses your terminology and handles your task without long instructions. Fine-tuning teaches behavior reliably; it teaches facts poorly.

Key takeaway: RAG changes what the model knows at query time; fine-tuning changes how it behaves. Match the mechanism to the problem before buying GPU time.

RAG vs fine-tuning: side-by-side comparison

Ranges are 2026 estimates for open-weight models on GPUs you own or rent.

Key takeaway: RAG wins on freshness, traceability and access control; fine-tuning wins on behavioral consistency and per-query cost at very high volume.

Decision checklist: which approach do you need?

Answer the questions in order and stop at the first clear yes.

1. Does the answer depend on information that changes weekly or faster (policies, prices, tickets)? Yes: RAG. Fine-tuned weights are stale the day training ends.
2. Must the answer show where it came from, for audit or user trust? Yes: RAG. Citations need retrieved passages.
3. Do users have different rights to the underlying data? Yes: RAG with permission-aware filtering. Never fine-tune on data some users may not see.
4. Does the model know the facts but answer in the wrong style, structure or length? Yes: fine-tuning, usually a LoRA adapter, once prompting with examples has failed.
5. Is the task narrow, repetitive and high-volume (classification, extraction, routing)? Yes: fine-tune a small model; it beats a large prompted model on cost and latency.
6. Does the model fail on domain language it has rarely seen (internal codes, a proprietary query language)? Yes: fine-tuning, with RAG for the facts behind that vocabulary.
7. Have you tried a structured system prompt with 5–10 examples on a current open-weight model? No: do that first. As of 2026, prompting plus RAG on Llama 4, Qwen 3 or Mistral resolves most enterprise cases; see which open-weight models fit enterprise use.
8. Still undecided? Build RAG first: it is reversible, cheap to try, and its logs become the training pairs you would need later.

Key takeaway: freshness, citations and access control force RAG; style, schema and volume argue for fine-tuning; everything else starts with prompting and RAG.

When to combine RAG and fine-tuning

The strongest production pattern is RAG for knowledge plus a LoRA adapter for behavior. Retrieval supplies current, cited facts; the adapter delivers your house style, schema and citation format without a long system prompt on every call. An adapter is small (tens to a few hundred megabytes) and the base model stays shared, so one vLLM deployment can serve many adapters and pick one per request.

Three variants recur: fine-tune the generator to use context well (cite, abstain when the documents are silent, ignore irrelevant passages); fine-tune the embedding model or reranker on your own query-document pairs, which often helps more than any change to the generator; or fine-tune a small model for structured output while a larger RAG pipeline handles open questions.

Key takeaway: combine them when facts and behavior both matter; keep facts in the index and behavior in the adapter, never the reverse.

RAG architecture essentials

A RAG pipeline is only as good as its retrieval. Most failures reported as "the model hallucinated" are retrieval failures: the right passage was missing from the top results or split so the relevant sentence lost its context.

Chunking

Split documents along their real structure (headings, sections, table rows), not at fixed character counts. Typical chunks are about 256–1,024 tokens with 10–20% overlap; smaller chunks retrieve precisely, larger ones preserve context. Store source, version, date, owner and access control list as metadata on every chunk. Parent-child chunking (retrieve small, return the parent section) suits policy and contract text.

Hybrid search

Combine dense vector search with a lexical index such as BM25 and merge the lists, typically with reciprocal rank fusion. Dense retrieval handles paraphrase; lexical retrieval handles product codes, part numbers, error strings and names that embeddings blur. Filter by metadata before ranking so access control and document versions are enforced in the retriever, not the prompt.

Reranking

Retrieve a broad candidate set (about 20–100 chunks), then apply a cross-encoder reranker to select the 3–10 passages that enter the prompt. Reranking is the cheapest quality gain in most pipelines because it reads query and passage together rather than comparing two independent vectors. NVIDIA NIM offers embedding and reranking microservices that run on the same GPUs as the generator.

Evaluation

Build a test set of 100–300 real questions with known source passages and expected answers before tuning anything. Measure retrieval (recall@k, MRR) separately from generation (faithfulness to context, answer correctness, citation accuracy). Re-run it on every change to chunking, embeddings or prompt; a change that improves generation but lowers recall is a regression you would otherwise meet in production.

Key takeaway: invest in chunking, hybrid search and reranking before touching the generator, and measure retrieval and generation separately.

Fine-tuning essentials

LoRA and QLoRA

LoRA freezes the base weights and trains low-rank matrices injected into the attention and MLP layers; the adapter is typically well under 1% of the model's parameters, a small file you can version and hot-swap. QLoRA holds the frozen base in 4-bit precision during training; the original paper fine-tuned a 65B model on a single 48 GB GPU. Full fine-tuning needs about 16 bytes per parameter for weights, gradients and optimizer state (roughly 1.1 TB for a 70B model), a multi-node job for cases where adapters are demonstrably insufficient.

As of 2026, an 8B-class model trains a LoRA adapter in hours on one H100 or RTX PRO 6000; a 70B model with QLoRA fits on one or two 80 GB GPUs at about a day per run. Training is rarely the bottleneck; data and evaluation are. Serving the adapter is a flag away in vLLM:

vllm serve /models/base-model \
  --enable-lora \
  --lora-modules house-style=/adapters/house-style \
  --max-lora-rank 64

Dataset size and quality

For style, format and tone, about 1,000–10,000 correct, consistent and representative examples outperform 100,000 scraped ones. Every example should be one you are happy to see reproduced, because it will be. Deduplicate, remove anything a user is not entitled to see, and hold out 10–20% for evaluation. Logged RAG interactions, corrected by domain experts, are usually the best source.

Evaluation and regression risk

Fine-tuning can degrade capabilities you did not train on: instruction following, refusals, other languages, tool calling and reasoning. Evaluate on three sets: the target task, a general capability suite, and your safety and compliance tests. Treat the adapter like a software release with a baseline, a diff and a rollback path. Pin the base model; a new base version invalidates the adapter and requires retraining and re-evaluation.

Key takeaway: use LoRA or QLoRA by default, spend most of the budget on data and evaluation, and gate every adapter behind regression tests.

Typical enterprise scenarios mapped to the right choice

Key takeaway: regulated and fast-changing use cases go to RAG; narrow, high-volume tasks go to fine-tuning; most complex products use both.

Frequently asked questions

Is RAG cheaper than fine-tuning?

Usually at the start: RAG needs no training run or labeled data, so a prototype costs days of engineering plus a vector database and an embedding model. Per request it costs more, because retrieved passages add about 2–10x more input tokens. Fine-tuning costs more up front but yields shorter prompts, so at very high volumes on a narrow task it is often cheaper overall.

Can fine-tuning teach an LLM new facts?

Only weakly and unreliably. A few thousand examples shift style and task behavior but do not create a dependable memory of specific facts, and the model cannot say where a fact came from. Continued pretraining on large domain corpora can add knowledge, but it costs far more and still gives no citations or updates. For facts that must be correct and current, use retrieval.

Do long context windows make RAG unnecessary?

No. As of 2026, open-weight models offer 128K-token context windows and more, enough to skip retrieval for a single small document set. For thousands of documents it does not scale: cost and latency grow with every token, accuracy drops as relevant passages are buried, and access control still has to happen somewhere. Retrieval selects what enters the window; long context lets you pass more of it.

How much data do I need to fine-tune an LLM?

For style, format and tone with LoRA, about 1,000–10,000 high-quality examples are typically enough, and quality matters more than count. A genuinely new skill or domain language needs more, often tens of thousands of examples. Whatever the size, deduplicate the data, review it for correctness, clear it for access rights, and hold out 10–20% for evaluation.

Does RAG eliminate hallucinations?

It reduces them substantially but not entirely. A grounded model instructed to answer only from retrieved context, and to abstain otherwise, makes far fewer unsupported claims. Errors remain when retrieval misses the right passage, when passages conflict, or when the model paraphrases loosely. Hybrid search, reranking, citation checks and a faithfulness metric in your evaluation set address most residual cases.

Can I fine-tune a model on confidential data safely?

Only with care. Training data can be reproduced by the model given the right prompt, and there is no reliable way to remove a document from trained weights. Fine-tune only on data every user of the model may see, strip personal data first, train and serve on infrastructure you control, and version the adapter so it can be withdrawn. Keep per-user confidential content in a permission-filtered RAG index.

How Nanobase AI can help

Nanobase AI designs, builds and operates both sides of this decision: permission-aware RAG pipelines (chunking, hybrid search, reranking, evaluation harnesses) and LoRA/QLoRA fine-tuning with regression suites, served on vLLM, TensorRT-LLM or NVIDIA NIM in your data center or private cloud. We size the GPUs (H100, H200, B200, RTX PRO), connect models to SAP, Salesforce, Microsoft 365 and Snowflake through MCP servers, and hand over runbooks your team can maintain.

Headquartered in Silicon Valley and a member of the NVIDIA Inception Program, we focus on insurance, finance and industrial teams that need answers they can trace and data that stays under their control. See our solutions for the full scope. Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.