A retrieval system that answers from your own documents is one of the highest value things you can build, and one of the easiest to build badly. The failure is rarely dramatic. It is a system that answers plausibly and is wrong often enough that people quietly stop trusting it.
Chunking is the highest leverage decision
Fixed size chunking with an arbitrary overlap is the default in every tutorial and it is wrong for most real document sets. It splits tables in half, separates a heading from the paragraph that gives it meaning, and cuts procedures at step four.
What works better, in order of how much difference it makes:
- Structure aware splitting that respects headings, sections, list boundaries, and table integrity
- Contextual headers prepended to each chunk, so a fragment carries the document title and section path with it
- Table extraction handled separately, preserving row and column relationships rather than flattening to text
- Chunk size tuned to your content type, not to a default. Dense policy documents want smaller chunks than narrative reports
Hybrid retrieval, not pure vector search
Semantic search is excellent at concept matching and unreliable at exact terms. Product codes, error identifiers, names, and version numbers are precisely the things users search for and precisely where embeddings are weakest.
Run keyword search alongside vector search and fuse the results. The improvement on real queries is consistently large, and it is the single cheapest quality gain available.
Rerank before you generate
Retrieve broadly, then rerank with a cross encoder and pass only the top few to the model. Feeding twenty chunks into a large context window and hoping the model picks the right one produces worse answers than feeding it four good ones, and it costs more.
More context is not better context. Precision at the top of the ranking is what determines answer quality.
Ground every claim, and say when you cannot
The answer should cite which source it came from, and the citation should be checkable. If the retrieved context does not support an answer, the correct output is to say so, not to produce a fluent guess.
This has to be enforced structurally rather than requested politely. Constrain the output format to require a source reference per claim, and validate that the referenced source was actually in the retrieved set.
How to measure it
Build an evaluation set before you build the pipeline. Fifty to two hundred real questions from your actual users, each with the correct answer and the document that contains it. Then measure three things separately:
| Metric | What it tells you | Fix if low |
|---|---|---|
| Retrieval recall at k | Whether the right document made it into the candidate set at all | Chunking, embedding model, hybrid search |
| Rerank precision at 3 | Whether the right chunk made it to the top after reranking | Reranker choice, chunk size |
| Answer faithfulness | Whether the generated answer is supported by what was retrieved | Prompt constraints, output schema, refusal behavior |
Measuring them separately matters because the fixes are different. A system with poor recall and a system with poor faithfulness look identical from the user's side and need completely different work.
The maintenance nobody plans for
Documents change. If the index is built once and never refreshed, the system confidently cites a policy that was superseded four months ago, which is worse than having no system.
Set up incremental reindexing on document change from day one. Retrofitting it is harder than it sounds because you need change detection you did not build.
A realistic quality target
On a well built system over a reasonable document set, expect to answer correctly on the large majority of in scope questions, and to correctly decline on most out of scope ones. Expect a residual band of ambiguous questions where the right answer depends on context the documents do not contain. Design the escalation path for that band rather than pretending it does not exist.
Written from work we have actually delivered. If your situation looks like the one described here, the quote form takes about two minutes and there is no sales sequence attached to it.
Get a free quote