How I Built a Production RAG System Answering 100K+ Documents at 94% Accuracy

A client had 100,000+ documents sitting in cloud storage and nobody could find anything in them. The knowledge was real. It was just unreachable. So in 2025, as an AI/ML Engineer at StechAI, I built a production RAG system on top of that corpus. It now answers questions at 94% retrieval accuracy, makes search 70% faster, and serves more than 10,000 predictions a day. This is the build, the numbers, and what I'd tell any founder weighing the same project.

What problem does a dormant 100K-document knowledge base actually cause?

The cost is invisible until you measure it. This client held 100,000+ documents that employees rarely opened, because keyword search returned noise instead of answers. People re-asked colleagues, re-wrote things that already existed, and made decisions without the relevant policy in front of them. The knowledge existed. It just didn't reach anyone.

That gap is common right now. McKinsey reports 72% of organizations already use generative AI, and 23% are scaling an agentic AI system ('The State of AI', McKinsey, 2025). Most of them are sitting on document piles like this one. The opportunity isn't a chatbot. It's turning dead archives into something people query in plain language and trust.

How is the production RAG system architected end to end?

The system answers in five stages: ingest, retrieve, rerank, generate, and cache. Each stage is a small, replaceable service, which is why it scaled to a 100K+ document corpus without a rewrite. I'll walk the pipeline in order, because the architecture is where 94% accuracy is won or lost, long before any model sees a token.

1. 1Ingestion: parsers normalize PDFs, docs, and HTML into clean text with source metadata preserved for citations.
2. 2Chunking: I split content semantically with overlap, so a chunk keeps enough context to stand on its own.
3. 3Embeddings + Qdrant: each chunk is embedded and stored in Qdrant, the vector database that handles similarity search at scale.
4. 4Hybrid retrieval: dense vector search runs alongside keyword search, then results merge so exact terms and meaning both count.
5. 5Cross-encoder reranking: a reranker re-scores the top candidates, pushing the genuinely relevant chunks to the top.
6. 6Generation: GPT-4 writes the answer from the reranked context and cites the source documents inline.
7. 7MCP agents: for multi-step questions, MCP agents break the query down, run sub-retrievals, and compose the final answer.
8. 8Redis caching: frequent queries and embeddings are cached, which is a big reason search runs 70% faster.

Here's the part people skip. The retriever is the product. GPT-4 and Claude only sound as smart as the chunks you hand them. If retrieval is wrong, the model writes a confident wrong answer. So I spent most of my time on the boring middle of the pipeline, not the model at the end.

What moved retrieval accuracy to 94%?

Accuracy went from mediocre to 94% through three changes, not one. Hybrid search came first, then cross-encoder reranking, then tighter chunking. None of these is exotic. Practitioners report hybrid search adds roughly 15-30% relevance, and cross-encoder reranking can lift it up to 48% on top of that. Stacked together, those gains compound fast.

Hybrid retrieval fixed the obvious failures first. Pure vector search missed exact identifiers, codes, and rare terms; pure keyword search missed paraphrased questions. Running both and merging scores caught both cases. The reranker then did the heavy lifting: it re-reads each candidate against the actual question and reorders them, so the model rarely sees a near-miss chunk dressed up as a match.

One more thing mattered: citations. Every answer points back to its source documents. That single feature is what turned skeptics into daily users, because they could verify instead of trust blindly.

What does the retrieval and generation code look like?

The core is small once the pipeline is in place. Below is a trimmed version of the LangChain RetrievalQA setup with the retriever configured for hybrid-style top-k search and source documents returned for citations. The interesting work lives in the retriever and reranker, not the chain itself.

from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI
from langchain_qdrant import QdrantVectorStore

vectorstore = QdrantVectorStore.from_existing_collection(
    collection_name="docs_100k",
    embedding=embeddings,
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 20})  # reranked down later

qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4", temperature=0),
    retriever=retriever,
    return_source_documents=True,  # powers inline citations
)

answer = qa.invoke({"query": user_question})

How does it serve 10K+ predictions a day reliably?

The system serves more than 10,000 predictions a day behind a FastAPI service, and it holds up because the slow parts are cached and the rest is async. FastAPI handles concurrency well, Redis absorbs repeat queries and embeddings, and Qdrant keeps similarity search fast at corpus scale. That combination is why search runs 70% faster than the keyword tool it replaced.

Monitoring is what keeps it honest. I track retrieval hit rate, answer latency, cache hit ratio, and token spend per query. When accuracy or latency drifts, I see it in the dashboard before a user complains. RAG quality decays quietly as documents change, so measuring it continuously is non-negotiable, not a nice-to-have.

The retriever is the product. The language model only sounds as smart as the chunks you hand it.

What actually worked in production?

A few decisions earned their keep, and I'd repeat all of them. These are the ones that moved metrics or saved me from a 2 a.m. incident, not the ones that looked good in a diagram. Here's the short list.

• Invest in retrieval before prompts. Hybrid search and a reranker beat any amount of prompt tuning.
• Return source documents and cite them. Verifiability is what got people to actually use it daily.
• Cache aggressively with Redis. Repeat queries are common, and caching is most of the 70% speed gain.
• Use MCP agents only for genuinely multi-step questions. Simple lookups don't need an agent.
• Measure retrieval quality continuously. It decays as the corpus changes, so set up monitoring on day one.

What were the hardest challenges and lessons?

The hardest problems weren't the models; they were the documents and the cost curve. Chunking a messy 100K+ document corpus took more iteration than anything else, and token spend crept up until caching and tighter retrieval brought it back down. The lesson held across the project: data quality and retrieval design decide success, and the LLM is the last 10%.

Demand for this kind of build is climbing too. Market analysts size the RAG market at roughly $2B in 2025 and project it toward about $10B by 2030 (MarketsandMarkets and Grand View Research estimates). My read after shipping one: the spend follows the same lesson. Teams that win put effort into ingestion and retrieval, not just into picking a bigger model.

1. 1Lesson one: bad chunks cause confident wrong answers. Fix chunking before blaming the model.
2. 2Lesson two: watch token cost from day one, because caching and reranking pay for themselves.
3. 3Lesson three: ship citations early. Trust is the adoption blocker, not raw accuracy.

Frequently asked questions

Do I need both GPT-4 and Claude in a RAG system?

No, one strong model is enough to start. I used GPT-4 for generation and Claude for parts of the agentic workflow because each had strengths for different steps. For most teams, pick one capable model, get retrieval right first, then add a second model only if a specific task clearly justifies it.

Why Qdrant instead of a managed vector service?

Qdrant gave me fast similarity search at 100K+ documents with full control over hosting and filtering. It handled hybrid retrieval cleanly and scaled with the corpus. A managed service can work too. The vector database matters less than your chunking and reranking, which is where retrieval accuracy is actually decided.

How long does a production RAG build like this take?

Most of the time goes to data, not modeling. A working prototype comes together quickly, but reaching 94% retrieval accuracy on a real 100K+ corpus took iteration on ingestion, chunking, and reranking. Plan for the data work to dominate the schedule, and budget time for monitoring before you call it production-ready.

Where should you start with your own corpus?

Start with the corpus you already have, not a bigger model. The biggest wins in this build came from hybrid retrieval, cross-encoder reranking, and clean chunking, all sitting in front of GPT-4 rather than inside it. With adoption now mainstream (McKinsey puts generative AI use at 72% of organizations in 2025), the edge is no longer access to models. It's building a production RAG system that retrieves the right context and proves it with citations. If you're sitting on a dormant knowledge base and want to talk through how to wake it up, I'm happy to discuss your project.

Sources

1. 1McKinsey, The State of AI 2025 - retrieved 2026-06-23
2. 2Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) - retrieved 2026-06-23
3. 3LangChain, RAG tutorial - retrieved 2026-06-23
4. 4OpenAI, Embeddings guide - retrieved 2026-06-23
5. 5Qdrant, Documentation - retrieved 2026-06-23