RAG quality is evaluated by measuring both the retrieval step and the generation step separately, and frameworks like RAGAS automate this by scoring a set of question-answer-context triples against several metrics without requiring hand-labeled ground truth for every metric. Faithfulness measures whether every claim in the generated answer is actually supported by the retrieved context, catching hallucinations even when the answer sounds plausible; context precision and context recall measure whether the retriever found the right chunks and whether it found all of them relevant to the question; and answer relevancy checks whether the generated response actually addresses what was asked rather than drifting off topic. RAGAS typically uses a strong language model as an automated judge to score these dimensions at scale, which is far faster than manual review but should be periodically spot-checked against human judgment to confirm the judge model is calibrated correctly. Running these metrics on a fixed evaluation set before and after any pipeline change, such as a new embedding model or chunking strategy, is what turns RAG tuning from guesswork into a measurable process. Beyond RAGAS, tools like TruLens and DeepEval offer similar metric sets with different integration options. Nanobase AI builds evaluation pipelines alongside every RAG deployment so retrieval and answer quality can be tracked over time.
A metric only matters if it runs before every change ships
RAGAS and similar frameworks are usually introduced as a one-time evaluation exercise, run once to get a baseline score and then forgotten. The larger value comes from wiring these metrics into a continuous pipeline that runs automatically whenever chunking, the embedding model, the retrieval logic, or the prompt template changes, so a regression is caught before it reaches production rather than discovered from a user complaint weeks later.
A RAG evaluation metric that runs once at launch and never again catches nothing about the regressions that happen after launch.
What RAGAS actually measures, split by pipeline stage
| Metric | Pipeline stage it evaluates | What a low score indicates |
|---|---|---|
| Context precision | Retrieval | Retrieved chunks contain irrelevant material mixed with relevant ones |
| Context recall | Retrieval | The retrieval step is missing chunks that contain the actual answer |
| Faithfulness | Generation | The generated answer contains claims not supported by the retrieved context |
| Answer relevancy | Generation | The generated answer does not actually address the question asked |
Separating these by pipeline stage is what makes the score actionable, since a low faithfulness score points at the prompt and model, not the retriever.
Building the pipeline in practice
- Maintain a versioned evaluation set of real or realistic question-context-answer examples, updated as new query patterns emerge from production logs.
- Run the full RAG pipeline against the evaluation set automatically on every pull request that touches chunking, retrieval, or prompt logic.
- Compute RAGAS metrics using an evaluator language model, and store results with the corresponding pipeline version for trend tracking over time.
- Set a minimum acceptable threshold per metric and fail the build if a change regresses below it, rather than only reviewing scores manually.
- Periodically sample real production queries into the evaluation set, since a static evaluation set gradually drifts from what users actually ask.
Treating RAG evaluation as a CI gate, the same way unit tests block a regression, is what turns RAGAS from a one-time report into an ongoing safeguard.
Where automated metrics diverge from human judgment
RAGAS metrics rely on an evaluator language model, which introduces its own imperfections, most notably that the evaluator can miss subtle factual errors it is not well equipped to catch, or penalize an answer that is correct but phrased differently than the reference. Automated scores are a strong early warning system for regressions and a good way to compare pipeline versions quickly, but a periodic human review of a sample of real outputs remains necessary to catch what the automated metrics systematically miss.
Automated metrics are efficient for catching regressions at pipeline-change speed; human review remains necessary for catching what an evaluator model misses.
Frequently asked questions
Does RAGAS require labeled ground truth answers?
Most RAGAS metrics, including faithfulness and answer relevancy, work without hand-labeled ground truth by using an evaluator language model to judge the generated answer against the retrieved context. Context recall benefits from a reference answer if one is available, improving its reliability.
How large does the evaluation set need to be?
A few dozen well-chosen questions covering common query patterns, edge cases, and known difficult queries is a reasonable starting point; the set should grow over time as new failure patterns are discovered in production.
Can RAGAS scores be compared directly across different corpora?
Not reliably. RAGAS scores are most useful for comparing pipeline versions against the same evaluation set over time, rather than as an absolute number comparable across unrelated projects or corpora.
What evaluator model should run the RAGAS scoring?
A capable instruction-following model is typically used as the evaluator, and using a different model family than the one generating answers reduces the risk of shared blind spots between the generator and the evaluator.
How Nanobase AI helps
Nanobase AI builds RAGAS-based evaluation into the CI pipeline for every RAG system we deploy, so chunking, retrieval, and prompt changes are measured against a versioned evaluation set before reaching production rather than after a user reports a bad answer. See our related guide on building a golden test set and solutions for the full delivery process.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.