A retrieval system does not fail dramatically. It fails politely - confident answers that are subtly wrong, or "I don't have that information" for a document you can see in the index. Both are usually the same three causes.
1. Your chunks split the answer in half
Fixed 500-token chunks cut mid-table and mid-procedure. The retriever then returns the half that mentions the keyword and drops the half with the actual number.
| Content type | Chunk strategy | Overlap |
|---|---|---|
| Policy docs, handbooks | Split on headings, keep the heading in every chunk | 10-15% |
| Tables, pricing sheets | One chunk per table, never split rows | 0% |
| Transcripts | Split on speaker turns, group into ~400-token windows | 1 turn |
| Code and API docs | One chunk per function or endpoint | 0% |
Rule of thumb: a chunk should be readable on its own by a human who has never seen the document. If it isn't, the model can't use it either.
2. Retrieval depth is guessed, not measured
`top_k = 5` is a default, not a decision. Run twenty real questions, log which chunk actually contained the answer, and look at its rank.
- Answer usually at rank 1-3 → keep k small, add a reranker only if precision drops.
- Answer often at rank 8-15 → your embedding model or chunking is the problem, not k. Raising k just buries the model in noise.
- Answer nowhere in the top 20 → it's a retrieval-coverage problem: missing document, bad OCR, or an acronym mismatch between question and source.
3. No citation rule, so nothing is checkable
Add one line to the system prompt and the failure mode changes from "invented answer" to "visible gap":
```
Answer only from the retrieved context below. Cite the source id in
brackets after each claim, e.g. [doc-14]. If the context does not
contain the answer, reply exactly: "Not in the knowledge base."
```
The exact-string escape hatch matters. Given permission to say a specific thing when it doesn't know, a model takes it far more often than when the instruction is a vague "don't make things up."
The evaluation set you can build in an hour
Write 20-30 real questions with the expected answer and the document that contains it. Then track three numbers on every change:
| Metric | What it catches |
|---|---|
| Retrieval hit rate | Did the right chunk make the top k at all? |
| Grounded answer rate | Did the answer come from the retrieved chunk, or from the model's memory? |
| Correct refusal rate | Does it say "not in the knowledge base" when it should? |
Without that third number, teams tune themselves into a system that never refuses and confidently answers questions it has no source for.
What to write down before you build
A workable RAG spec is one page: sources and refresh cadence, chunking rule per source type, embedding model, retrieval depth and reranking, the citation and refusal contract, and the eval set. Build from the spec, not from the SDK quickstart.
Start with the [Knowledge Assistant](/knowledge-assistant) tool if you want that page generated from your own inputs.