Hybrid retrieval: why pure vector search fails in production
Retrieval-augmented generation looks simple in a notebook: embed the documents, embed the question, return the nearest neighbours, hand them to the model. Then it meets a real corpus, and the answers quietly get worse. Here is what we change when a prototype has to become a system people rely on.
Pure vector search fails in predictable ways
Dense embeddings capture meaning, which is exactly what you want when a user asks about a "quarterly revenue slump" and the document says "Q3 income decrease". They are much weaker at the things business users actually type: order numbers, part codes, error codes, contract references, surnames.
Ask a dense index for error E-4099-X and it will happily return five documents about error codes in general, none of them the right one. The nearest neighbour in embedding space is not the exact match the user needed, and the model has no way to know the difference.
Hybrid retrieval, then reranking
The fix is not exotic. Run two retrievers and combine them:
- Dense vector search for semantic intent, so paraphrases and synonyms still match.
- Keyword search (BM25) for exact terminology: identifiers, codes, names, quoted phrases.
- Fusion of both result sets, usually reciprocal rank fusion, which needs no score calibration between the two systems.
- A reranker over the top 30 to 50 candidates, which is where most of the quality comes from and where a cross-encoder earns its latency.
In practice the reranker matters more than the embedding model. Teams spend weeks comparing embeddings and skip the step that reorders candidates using the question and the passage together.
Chunking is a product decision
Chunk size is usually treated as a tuning parameter. It is closer to a product decision: it decides what a citation looks like. If a chunk spans three sections, the user gets a citation they cannot verify at a glance.
We chunk on document structure rather than a fixed token count, keep headings with their content, and store the section path as metadata. That metadata does double duty: it lets users filter by product area or document type, and it gives the answer somewhere concrete to point at.
Make the model cite, then check it
An answer without a verifiable citation is a guess with good grammar.
We require every factual sentence to carry a chunk id, and we validate those ids after generation. If a citation is missing or points at a chunk that was never retrieved, the response does not go out: the system falls back to showing the retrieved passages and an honest "I could not answer this from the documents" message.
Users forgive a system that says it does not know. They do not forgive one that invents a policy and attributes it to the company handbook.
Evaluate on your own questions
Public benchmarks will not tell you whether your retrieval works. Build a small evaluation set from real questions, a hundred is enough to start, and track two numbers separately:
- Retrieval recall: was the passage containing the answer in the candidate set at all?
- Answer quality: given the right passages, did the model use them correctly?
Keeping these apart tells you where to spend your time. Most teams that believe they have a model problem have a retrieval problem, and no amount of prompt engineering fixes a passage that was never retrieved.
What we would do first
If you are moving a RAG prototype toward production, in order: add keyword search alongside your vector index, add a reranker, build an evaluation set from real questions, then enforce citations. That sequence has fixed more retrieval quality problems for us than any change of model.
