A golden test set for RAG is a curated collection of representative questions paired with their correct answers and, ideally, the specific source chunks that should be retrieved to answer them correctly, used as a fixed benchmark to measure whether pipeline changes actually improve or hurt real-world accuracy. Building one typically starts by collecting real questions from support tickets, chat logs, or subject-matter experts rather than inventing hypothetical ones, since real questions capture the actual phrasing and edge cases the production system will face. Each question should be paired with a verified correct answer and, where possible, an explicit list of the document chunks that contain the supporting information, which allows retrieval metrics like context precision and recall to be measured directly rather than only judging the final generated answer. A useful golden set covers easy factual lookups, harder multi-part questions, cases the corpus genuinely cannot answer, and adversarial phrasing that differs from document wording, typically totaling fifty to a few hundred examples depending on domain complexity. Running the full RAG pipeline against this set before and after any change, such as a new chunking strategy or embedding model, turns tuning decisions into measurable comparisons instead of subjective impressions. Nanobase AI builds a golden evaluation set as one of the first deliverables in every RAG engagement.
A test set is only useful once it is scored against real metrics
Collecting a set of representative questions and correct answers is the easy half of building a golden evaluation set; the harder half is choosing metrics that catch the failure modes that actually matter and running them consistently every time the pipeline changes. A golden set that exists as a spreadsheet nobody re-runs after the first tuning pass provides no more protection against regression than having no test set at all. A golden test set only prevents regressions if it is actually re-run on every meaningful pipeline change, not just once during initial tuning.
The metrics that matter for RAG specifically
| Metric | What it measures | What it catches |
|---|---|---|
| Context precision | Share of retrieved chunks that are actually relevant | Noisy retrieval diluting the model's context |
| Context recall | Share of needed information that was retrieved at all | Missing chunks the answer depended on |
| Faithfulness | Whether the generated answer is supported by retrieved context | Hallucination or the model going beyond its sources |
| Answer relevancy | Whether the answer actually addresses the question asked | Off-topic or evasive responses despite correct retrieval |
| Citation accuracy | Whether cited sources actually contain the claimed information | Citations that look plausible but are wrong |
Key takeaway: context precision and recall isolate retrieval quality, while faithfulness and answer relevancy isolate what the model did with what it retrieved, and both pairs need tracking separately.
Building the evaluation loop into deployment
- Store the golden set as version-controlled data, not a one-off spreadsheet, so it evolves alongside the product and every change to it is reviewable like a code change.
- Run the full pipeline, retrieval and generation together, against the golden set automatically whenever a change touches chunking, embeddings, the vector database configuration, the reranker, or the prompt template.
- Score each run against the metrics above and compare to the previous baseline, flagging any metric that regresses beyond an agreed threshold rather than relying on a person eyeballing sample outputs.
- Block or flag deployment of a change that regresses a core metric, the same way a failing test suite blocks a code merge, so quality regressions are caught before reaching production users.
- Periodically add new real questions from production logs and support tickets to the golden set, since a static set stops reflecting how the system is actually being used within a few months.
Key takeaway: treating RAG evaluation as a gate in the deployment pipeline, not a manual spot-check, is what actually prevents shipping a regression.
Sizing and composing the set correctly
A golden set that is too small or too easy gives a false sense of confidence, while one that is enormous slows down the iteration loop enough that teams stop running it regularly. A composition of fifty to a few hundred examples, weighted toward the query patterns seen most often in real usage but deliberately including harder multi-part questions, edge cases the corpus genuinely cannot answer, and adversarial phrasing that differs from the document's own wording, gives a more honest signal than an easy set drawn entirely from FAQ-style questions. Including a handful of unanswerable questions matters specifically because a system that always attempts an answer, even to questions the corpus has no information about, is a distinct and dangerous failure mode that only shows up if the test set includes cases designed to trigger it.
Key takeaway: deliberately include unanswerable and adversarial questions in the golden set, since a system that never says "I don't know" is failing a case that easy questions will never reveal.
Frequently asked questions
How often should the golden set be updated?
Add new examples whenever a production issue reveals a case the set did not cover, and do a broader review quarterly to ensure the set still reflects current document content and real query patterns, since a corpus and its users both drift over the months a system stays in production.
Can this evaluation be automated without human review of every answer?
Yes, using an LLM-as-judge approach where a separate model scores faithfulness and relevancy against the retrieved context and the golden answer, though this should be periodically spot-checked against human judgment to confirm the judge model itself is scoring reliably rather than introducing its own bias.
What is a reasonable threshold for blocking a deployment?
There is no universal number, but many teams block on any regression larger than a few percentage points on faithfulness or context recall specifically, since these two metrics most directly predict whether users will see a wrong or ungrounded answer, while allowing smaller fluctuations on softer metrics like answer relevancy.
Does a golden test set replace the need for production monitoring?
No. A golden set catches regressions before deployment on known question types; production monitoring, including user feedback signals and sampled live query review, catches drift and new failure patterns that the fixed golden set was never designed to cover.
How Nanobase AI helps
Nanobase AI builds a golden evaluation set with these metrics as one of the first deliverables in every RAG engagement, then wires it into the deployment pipeline as a genuine quality gate rather than a one-time report. This is the same evaluation discipline we apply when diagnosing an underperforming RAG system. See our solutions for the full pipeline.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.