Retrieval Is a Data Problem, Not a Vector Problem

Every team that ships a retrieval-augmented feature goes through the same arc. The prototype is astonishing. The pilot is fine. Then real users arrive with real questions and the answers get vague, subtly outdated, or confidently wrong about the one document that mattered. The instinct is to reach for a better embedding model or a different vector database. That is almost never where the problem is.

Your chunks are the product

Retrieval quality is bounded by whether a single retrieved chunk can actually answer the question. Splitting documents into fixed windows of a thousand characters guarantees that half your chunks begin mid-sentence and end mid-table, and that the heading explaining what the section is about lives in a different chunk from the content.

The fix is structural, not statistical. Split on document structure, keep the heading path as a prefix on every chunk, and never let a table be severed from the sentence that defines its columns. In practice, prepending a breadcrumb such as the document title and section heading to each chunk does more for answer quality than any embedding upgrade I have measured.

Stale data beats wrong data at looking correct

A retrieval system with no freshness signal will happily quote a policy that changed four months ago, with a citation, in a confident tone. Users trust it precisely because it cites a source. This is the most damaging failure mode in the entire category, because it is invisible until someone acts on it.

Two mitigations, both cheap. Store an updated timestamp with every chunk and surface it in the answer, so the reader can judge. And build a reindex path that is triggered by writes rather than by a nightly cron, so the window in which the index disagrees with the source of truth is minutes rather than a day.

Permissions cannot be a post-filter

If you retrieve first and filter by access afterwards, two things go wrong. Users get thin answers because their top ten results were mostly documents they cannot see, and eventually someone leaks a snippet through a summary. Access has to be a condition of the search itself: filter by the user's permitted set inside the query, not after it. This is unglamorous plumbing, and it is the difference between a demo and something legal will let you ship.

Hybrid search is not optional

Pure vector search is weak exactly where enterprise questions live: product codes, error identifiers, invoice numbers, names of internal tools. Semantic similarity does not help you find ERR_5521; exact matching does. Run keyword and vector retrieval together, merge the results, and rerank. The lift from adding plain keyword search alongside embeddings is usually larger than the lift from any model swap.

Measure retrieval separately from generation

When an answer is bad, you need to know which half failed. Build a small labelled set that maps questions to the documents that genuinely contain the answer, then measure whether the right document appears in your top results at all. If recall is poor, no prompt engineering will save the output. If recall is good and answers are still bad, the problem is in synthesis and you should look at your prompt, your context ordering, or your instruction to abstain.

Teaching a system to say "I could not find this" is worth more than another two points of retrieval accuracy. Abstention is a feature, and it has to be trained for and rewarded.

An order of operations that works

1. Fix chunking so each chunk is self-describing.
2. Add keyword retrieval alongside vectors and rerank the union.
3. Make permissions part of the query.
4. Attach and display freshness.
5. Build the labelled recall set and measure.
6. Only then consider changing your embedding model.

Most teams do this list backwards, starting with step six because it feels like the AI part. The unglamorous first five are where the answers actually come from.