MCP and RAG in production: connecting LLMs to your enterprise data the right way

01RAG, beyond the toy version

The tutorial version of RAG - embed some documents, stuff the top matches into the prompt - works in a demo and falls apart in production. Real RAG is a retrieval pipeline: chunking that respects document structure, hybrid search that combines vector similarity with keyword and metadata filters, a reranking pass to push the genuinely relevant context to the top, and a generation step that is forced to cite its sources.

The single highest-leverage change most teams can make is reranking. Vector search alone returns plausible-looking chunks; a reranker returns the right ones, and answer quality jumps without touching the model.

• Structure-aware chunking - don't split tables, code or clauses mid-thought.
• Hybrid retrieval - vector + keyword + metadata filters, not vectors alone.
• Reranking - the cheapest large quality win available.
• Grounded generation - cite sources, and say 'I don't know' when retrieval is empty.

02Where MCP fits

RAG is great for knowledge that lives in documents. It is the wrong tool for live state - the current order status, today's inventory, a customer's open tickets. That is what the Model Context Protocol is for: a standard way to expose tools, resources and data sources to a model through a well-defined server interface, so the model can fetch live context or take an action instead of guessing from stale text.

The architectural win is decoupling. Your MCP servers wrap each system once, with their own auth and permissions, and any model or agent can use them through one protocol. You stop writing bespoke glue for every model-to-system connection.

03Security is the whole game

Connecting a model to enterprise data is a data-governance decision before it is an engineering one. Retrieval must respect the asking user's permissions - never embed a document into a shared index that some users shouldn't see. MCP servers must enforce authorization on every call, because a model will happily request anything. And both layers must redact sensitive fields before content ever reaches the prompt.

Treat retrieved content and tool outputs as untrusted. The same prompt-injection risk that affects agents applies here: a poisoned document can carry instructions. Isolate retrieved text from system instructions, and never let retrieved content silently change what the model is allowed to do.

04Measuring whether it works

Ground truth matters. Maintain an evaluation set of real questions with known-good answers and known-good sources, and score both answer accuracy and retrieval precision separately - because a wrong answer from correct context is a generation problem, while a wrong answer from missing context is a retrieval problem, and they need different fixes. Without that split, teams spend weeks tuning the wrong layer.