RAG That Actually Retrieves: Lessons From a £6B Trust Platform

I built a RAG knowledge bot inside a trust-management platform that moves over £6 billion a year. The bot answers staff questions against a large corpus of internal policy, procedure, and compliance documents. The first version — OpenAI embeddings, Pinecone, cosine top-k — demoed beautifully and failed in production. The version that shipped improved document retrieval by 45%. The gap between those two is entirely in the unglamorous parts.

Here's what actually moved the needle.

Naive cosine top-k fails on enterprise docs

The seductive RAG tutorial is: embed your docs, embed the query, return the top-k nearest by cosine similarity, stuff them in the prompt. It works great on clean Wikipedia-style passages. It falls apart on enterprise documents, and it's worth being precise about why.

Enterprise docs are dense with near-synonyms and exact identifiers. A user asks about "the £10k approval threshold for vendor onboarding." The relevant passage says "supplier authorization limit." Semantically close, lexically disjoint — and the one token that actually disambiguates the answer (£10,000) is a literal string that embeddings handle poorly. Dense vectors are great at meaning and bad at exact matches. Policy documents are full of exact matches that matter: section numbers, monetary limits, defined terms.

The second failure is chunk boundaries destroying meaning. A naive 500-token sliding window will split a numbered procedure right down the middle, so the chunk that ranks highest contains steps 4 through 7 with no idea what the procedure is for.

Chunking: structure-aware, not fixed-window

The single highest-leverage change was abandoning fixed-size chunks for structure-aware ones. These documents have real structure — sections, sub-clauses, numbered steps — so we chunk on those boundaries and never split a logical unit. Each chunk carries enough of its parent heading to stay self-describing, and we attach rich metadata: document type, section path, effective date, and which business unit owns it.

That metadata is not decoration. It's what makes the next two improvements possible.

Hybrid retrieval: dense + sparse, then filter

We stopped relying on cosine alone and went hybrid: dense vectors for semantic recall, sparse (BM25-style keyword) for exact-term precision, fused into one ranking. The dense side finds "supplier authorization limit" from a query about "vendor approval." The sparse side guarantees that when someone types £10,000 or Section 4.2, the chunk containing that literal token is in contention. Neither alone is sufficient; together they cover each other's blind spots.

Then we filter on metadata before scoring, not after. If a user is asking about a current procedure, there's no reason superseded 2019 revisions should compete for the top-k slots. Pinecone applies the filter at query time, which both improves relevance and shrinks the candidate set the ranker has to reason over.

results = index.query(
    vector=dense_embedding,
    sparse_vector=sparse_embedding,   # hybrid: dense + sparse fused
    top_k=8,
    filter={
        "doc_type": {"$in": ["policy", "procedure"]},
        "status": {"$eq": "active"},        # drop superseded revisions
        "effective_date": {"$lte": today},  # nothing not yet in force
    },
    include_metadata=True,
)

Filtering pre-scoring is the cheapest accuracy win in the whole pipeline. You're not making retrieval smarter — you're refusing to let irrelevant candidates dilute the ranking in the first place.

MCP: let the model call retrieval as a tool

The other structural decision was not stuffing retrieved context into a single mega-prompt. Instead we exposed retrieval as a tool over MCP and let the model call it — sometimes more than once, with refined queries — as part of answering.

This matters because a single retrieval pass assumes the user's literal question is the best query. It often isn't. A vague question ("what's our process for high-risk vendors?") retrieves better after the model has reformulated it into the platform's actual vocabulary, or after it's seen one chunk and realized it needs the referenced sub-procedure too. Tool-calling lets the model iterate: retrieve, read, decide it needs more, retrieve again with a sharper query. Context-stuffing can't do that — it's one shot, and it wastes the window on chunks that turned out to be irrelevant.

The MCP tool definition is small and the description does real work — the model reads it to decide when and how to call:

import { Server } from "@modelcontextprotocol/sdk/server";
 
server.tool(
  "search_knowledge_base",
  {
    description:
      "Search internal policy and procedure documents. Use for any question " +
      "about company processes, approval thresholds, or compliance rules. " +
      "Pass a focused query; you may call this multiple times to refine.",
    inputSchema: {
      type: "object",
      properties: {
        query: { type: "string", description: "Focused search query" },
        doc_type: {
          type: "string",
          enum: ["policy", "procedure", "any"],
          description: "Narrow to a document class when known",
        },
      },
      required: ["query"],
    },
  },
  async ({ query, doc_type }) => {
    const hits = await hybridRetrieve(query, {
      filter: doc_type === "any" ? {} : { doc_type },
      topK: 8,
    });
    return {
      content: hits.map((h) => ({
        type: "text",
        text: `[${h.metadata.section_path}] ${h.text}`,
      })),
    };
  },
);

Stop thinking of retrieval as a preprocessing step and start thinking of it as a capability the model invokes. Once retrieval is a tool, the model can ask better questions than the user did.

Note the section_path prepended to each result. The model gets to see where a chunk lives in the document hierarchy, which both improves its answers and lets it cite precisely — "per Procedure 4.2" instead of a vague paraphrase. In a compliance context, a citable answer is worth far more than a fluent one.

What added up to +45%

No single trick did it. The 45% retrieval improvement came from stacking changes that each fix a specific failure of naive top-k:

• Structure-aware chunking — stop splitting logical units; keep chunks self-describing.
• Hybrid dense + sparse — semantic recall and exact-identifier precision.
• Metadata filtering pre-scoring — kill superseded and out-of-scope candidates before they dilute the ranking.
• MCP tool-calling — let the model retrieve iteratively instead of betting everything on one prompt-stuff.

The stack — OpenAI, LangChain, MCP, Pinecone, Node.js — is unremarkable. The result wasn't, because the work went into matching the retrieval strategy to what enterprise documents actually are: structured, identifier-heavy, versioned, and unforgiving of the demo-grade defaults.