Muhammad Noman RiazYou cannot improve what you do not measure. I learned that the hard way shipping retrieval-augmented generation to production. The model looked fine in demos, then fell apart on real questions, and I had no numbers to tell me why. Here is the uncomfortable truth most teams discover late: the majority of RAG bugs hide in retrieval, not generation. The LLM is usually doing its job. It is just being handed the wrong context. This is a hands-on guide to evaluating a RAG system the way I do it now, after holding 94% retrieval accuracy across 100K+ documents in production serving 10K+ predictions a day.
Key Takeaways
- Most RAG failures are retrieval failures, so measure retrieval and generation separately.
- Build a small golden eval set early, then score it with RAGAS in CI on every change.
- I held 94% retrieval accuracy over 100K+ documents by baking evaluation in from day one.
- Monitor retrieval drift weekly in production. Offline scores go stale as your corpus and queries shift.
Why is evaluation the step most RAG teams skip?
Evaluation gets skipped because demos feel like proof, and they are not. A handful of cherry-picked questions hide the long tail where systems break. In my experience, teams ship on vibes, then spend weeks firefighting blind. Without metrics, every fix is a guess and every regression is invisible until a user complains.
The cost compounds fast. When you change a chunk size, swap an embedding model, or tweak a prompt, you have no way to know if you helped or hurt. I have watched a single 'small' chunking change quietly drop answer quality for a whole category of questions. Nobody noticed for two weeks because nobody was measuring.
Most RAG bugs are not generation bugs. They are retrieval bugs wearing a generation costume. If you only read the final answer, you will misdiagnose almost every failure.
Why separate retrieval metrics from generation metrics?
Separate them because they fail for different reasons and need different fixes. Retrieval metrics tell you whether the right context reached the model. Generation metrics tell you what the model did with that context. Blend them into one number and you lose all diagnostic power. I keep two scoreboards so I always know which half of the pipeline is bleeding.
Here is the mental model I use. If retrieval is broken, generation cannot recover, because the answer is being written from missing or irrelevant context. So I check retrieval first. Recall@k and precision@k tell me if the chunks are there and clean. Context relevance tells me how on-topic they are. Only once retrieval is solid do faithfulness and answer relevance become meaningful signals.
| Metric | What it measures | Stage | Watch for |
|---|---|---|---|
| recall@k | Whether the relevant chunks appear in the top-k results | Retrieval | Low recall means the answer was never retrievable. Raise k or fix chunking. |
| precision@k | How many of the top-k chunks are actually relevant | Retrieval | Low precision floods the prompt with noise and distracts the model. |
| context relevance | How on-topic the retrieved context is to the query | Retrieval | Drops first when embeddings or the corpus drift. An early warning sign. |
| faithfulness | Whether the answer is grounded in the retrieved context | Generation | Low faithfulness means hallucination. Good context, bad grounding. |
| answer relevance | Whether the answer actually addresses the question asked | Generation | Can stay high while faithfulness is low. Confident, fluent, and wrong. |
How do you build a golden eval set first?
Start small and start by hand. A golden eval set is a fixed list of real questions paired with the correct context and a reference answer. I began with about 50 questions pulled from actual user logs, not invented ones. That tiny set caught more regressions than any synthetic benchmark I tried later, because it reflected how people really asked.
- 1Pull real questions from logs or support tickets, including the awkward and ambiguous ones.
- 2For each question, record the chunk(s) that should be retrieved as ground truth.
- 3Write or approve a reference answer so faithfulness and relevance have something to anchor to.
- 4Cover your hard categories on purpose: multi-hop questions, edge cases, and known weak spots.
- 5Version the set in git and grow it every time a real bug slips through.
Do not wait for perfection. Fifty good examples beat zero examples, and a flawed eval set you actually run beats a pristine one you keep meaning to build. I treat the golden set as living: every production failure becomes a new test case, so the same bug can never embarrass me twice.
How do you score it with RAGAS?
I use RAGAS because it scores retrieval and generation in one pass with metrics that map cleanly to my two scoreboards. It computes faithfulness, answer relevance, and context precision from your questions, retrieved contexts, and answers. The setup is short. Once it runs, you have a repeatable number instead of a gut feeling about every change you make.
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
)
# Each row: the question, your system's answer,
# the retrieved chunks, and the reference answer.
eval_data = Dataset.from_dict({
"question": questions,
"answer": answers,
"contexts": retrieved_contexts, # list[list[str]]
"ground_truth": reference_answers,
})
result = evaluate(
eval_data,
metrics=[faithfulness, answer_relevancy, context_precision],
)
print(result) # per-metric scores you can track over timeRead the scores together, not in isolation. High answer relevance with low faithfulness is the classic trap: a fluent answer that the context does not support. Low context precision usually points back at retrieval, so I fix that before I touch the prompt. The point is not a single grade. It is knowing which lever to pull.
Why should evals run in CI, not just once?
Run evals in CI because RAG systems regress silently and constantly. A new embedding model, a reindex, a prompt edit, a dependency bump: any of these can quietly degrade quality. A one-time evaluation tells you how things looked that afternoon. A CI gate tells you the moment a change makes things worse, while the change is still fresh in your head.
My setup is simple. On every pull request that touches retrieval, prompts, or indexing, CI runs the golden set through RAGAS and prints the scores. I set thresholds on faithfulness and context precision, and if a metric drops below the floor, the build fails. It feels strict at first. After it catches its first real regression, nobody on the team wants to remove it.
Offline evals in CI catch the regressions you cause. Production monitoring catches the regressions the world causes. You need both, because your code and your data drift on different clocks.
How do you monitor retrieval drift in production?
Offline scores go stale the moment real traffic shifts, so I monitor retrieval in production continuously. This is exactly how I held 94% retrieval accuracy across 100K+ documents at 10K+ predictions a day. The corpus grew, user questions changed, and context relevance started sliding before any user thought to complain. Watching that one signal gave me a head start on every drift event.
Retrieval drift is sneaky because nothing errors out. The pipeline keeps returning chunks, they are just slowly getting less relevant as new documents crowd the index and query patterns evolve. I log retrieval scores on live traffic, sample real queries for human review, and trend context relevance week over week. When the trend bends down, I reindex or retune to improve accuracy before it ever shows up in a complaint.
- Track context relevance trends on live traffic week over week, not just the daily average.
- Sample a set of real production queries and review retrieved chunks by hand each week.
- Watch the rate of low-confidence or empty retrievals as a leading drift signal.
- Compare this week's score distribution against your golden-set baseline.
- Flag new high-frequency query types your index may not cover yet.
- Re-run the full golden set after every reindex and log the result.
Frequently asked questions
How big should my golden eval set be to start? Smaller than you think. I started with around 50 real questions and got real signal immediately. Quality and coverage of your hard cases matter far more than raw count. Grow it as production surfaces new failure modes, and let every escaped bug earn its own permanent test case.
Do I really need RAGAS, or can I score by hand? Hand scoring works for a tiny set, but it does not scale and it is not repeatable. RAGAS gives you consistent faithfulness, answer relevance, and context precision numbers you can gate CI on. I still keep humans in the loop for sampling, but automated scoring is what makes evaluation a habit instead of a heroic one-off.
What should I fix first when scores look bad? Fix retrieval first, almost always. If context precision or relevance is low, the model is working from bad source material and no prompt tweak will save it. Get the right chunks into the prompt, then judge faithfulness and answer relevance. Diagnosing in that order has saved me from countless wild goose chases.
Where to go from here
Evaluation is not a phase you finish. It is the instrument panel you fly your RAG system with. Separate retrieval from generation, build a small golden set from real questions, score it with RAGAS in CI, and watch retrieval drift in production. That loop is how I reached and held 94% retrieval accuracy across 100K+ documents under real load. None of it is exotic. It is just measured, on purpose, all the time. And if your scores stay stubbornly low no matter how you tune retrieval, that data is also what tells you whether to keep iterating on RAG or fine-tune instead.
If you are shipping RAG and flying blind, start this week. Pull 50 real questions, wire up RAGAS, and add one CI threshold. You will catch your first regression faster than you expect. If you want a second set of eyes on your evaluation setup or your retrieval pipeline, reach out and let's talk through it.
Sources
- 1RAGAS, Documentation - retrieved 2026-06-23
- 2Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (2023) - retrieved 2026-06-23
- 3TruLens, Documentation - retrieved 2026-06-23
Get the next post
New writing on AI engineering, RAG, and shipping real products - straight to your inbox, no noise.