Fixing a RAG system that gives wrong answers requires a partner who can diagnose which stage of the pipeline is actually failing, since chunking, embedding model choice, retrieval method, reranking, and prompt construction are all common failure points and the fix looks completely different depending on which one is broken. A capable diagnostic process starts by building a labeled evaluation set of real questions with known correct answers and source chunks, then measuring retrieval precision and recall separately from generation quality, so the team can tell whether the retriever is failing to find the right chunks or the language model is failing to use correct chunks properly once retrieved. Common fixes include adding hybrid search and reranking where vector search alone is missing exact-match queries, restructuring chunking around document sections rather than fixed character counts, tightening the prompt to constrain the model to retrieved context only, or switching to an embedding model better suited to the corpus's language or domain. Vendors who jump straight to trying a different language model without first measuring retrieval quality are treating a likely retrieval problem as a generation problem, which rarely fixes the underlying issue. Nanobase AI runs a structured diagnostic against a labeled evaluation set before recommending changes to an underperforming RAG system, rather than guessing at a fix.
The engagement should look like an audit, not a rebuild
When a RAG system is giving wrong answers, the instinct is often to ask a vendor to "fix the RAG system," but a capable partner should push back on that framing before starting work, because a wrong-answer system usually needs a structured audit to find the actual failing stage before any fix makes sense. An engagement that starts with changes, such as swapping the language model or adding a reranker, before establishing where the pipeline actually breaks is guessing with billable hours. A RAG accuracy engagement should start with diagnosis, not changes, and a partner who proposes fixes before measuring where the pipeline fails is skipping the step that makes the fix reliable.
What the phases of a real fix engagement look like
| Phase | Typical duration | Output |
|---|---|---|
| Discovery and instrumentation | Days to one to two weeks | A labeled evaluation set built from real failing examples, plus retrieval and generation metrics baselined |
| Stage-by-stage diagnosis | One to two weeks | A specific finding: chunking, embedding mismatch, missing hybrid search, weak reranking, or prompt construction identified as the primary cause |
| Targeted fix implementation | Two to four weeks, depending on scope | Changes to the identified stage, re-tested against the same labeled set before and after |
| Re-evaluation and sign-off | Days to one week | Measured improvement on the same metrics used at baseline, not a new, more favorable test set |
| Handoff or ongoing monitoring | Ongoing | Dashboards and an evaluation process the internal team can run independently going forward |
Key takeaway: a legitimate fix engagement measures the same metrics before and after the change, on the same test set, so improvement is demonstrated rather than asserted.
Why root-cause diagnosis changes the fix completely
The specific cause behind wrong answers determines a completely different remediation path, which is exactly why skipping diagnosis wastes effort: if chunking is splitting answers from their context, the fix is restructuring how documents are split, unrelated to the language model at all; if retrieval is missing exact-match queries like product codes, the fix is adding hybrid search, not touching the embedding model; if the correct chunks are retrieved but the model still answers incorrectly, the fix is in prompt construction, and neither a different chunking strategy nor a different vector database would help at all. A partner that proposes the same generic fix, such as "switch to a bigger model," regardless of the diagnosis, has not actually diagnosed anything, which is a useful signal when evaluating a proposal. This is the same stage-by-stage logic covered in why RAG returns irrelevant results, applied here as a structured engagement rather than a self-service checklist.
Key takeaway: the correct fix depends entirely on which stage is failing, so a proposal that recommends the same change regardless of diagnosis has likely skipped the diagnosis.
What good evidence of improvement looks like
A credible fix engagement reports improvement in terms that map to the original problem: a specific increase in context recall for the question types that were previously failing, a reduction in the rate of ungrounded or hallucinated claims measured by faithfulness scoring, or a measurable drop in the rate of user-flagged wrong answers in production after deployment. Vague claims of "significantly improved" without a metric, a baseline, and a comparable test set should be treated skeptically, since the entire point of instrumenting the pipeline at the start of the engagement is to make this comparison possible and specific rather than subjective.
Key takeaway: insist on a before-and-after metric on the same test set as proof of improvement, not a qualitative claim that the system feels better.
Frequently asked questions
How long does a typical fix engagement take?
A focused diagnosis phase often takes one to two weeks, and the full engagement including implementation and re-evaluation commonly runs four to eight weeks depending on how many stages need remediation and how complex the underlying document set is, though this varies with the scope of the actual findings.
Should we build our own evaluation set before hiring help?
It helps but is not required, since a capable partner will build or refine one as part of discovery. Bringing real examples of wrong answers your team has already collected, even informally, speeds up the diagnosis phase considerably compared to starting from a completely generic test set.
Can this be fixed without touching the language model at all?
Often yes. Many accuracy problems trace back to chunking, missing hybrid search, or weak reranking rather than the language model itself, and fixing those stages frequently resolves the issue without any change to which model generates the final answer.
What if the diagnosis finds multiple contributing causes?
This is common, and a structured engagement should prioritize fixes by expected impact, addressing the cause responsible for the largest share of failures first and re-measuring before moving to the next, rather than changing everything simultaneously and being unable to attribute the improvement to any specific fix.
How Nanobase AI helps
Nanobase AI runs a structured diagnostic against a labeled evaluation set before recommending any change to an underperforming RAG system, then measures improvement against the same baseline once a fix is implemented. This audit-first approach is the same discipline we bring to new RAG builds from scratch. See our solutions for the full scope.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.