Retrieval or a bigger context window: how to actually decide

Context windows are now large enough that pasting an entire handbook into a prompt genuinely works. That has made a lot of retrieval pipelines unnecessary — and a lot of others suddenly look like premature engineering. The decision is not ideological. It comes down to how much text you have, how often it changes, and who is allowed to see it.

Put it in the prompt when

• The whole corpus fits comfortably and is not growing much — a policy document, a product spec, a contract under review.
• Everyone asking is allowed to see everything. No per-user permissions to enforce.
• The content changes rarely, or you can afford to resend it every time.

In that world a retrieval layer is machinery you have to maintain for no benefit.

Build retrieval when

• The corpus is large, and most of it is irrelevant to any given question. You are paying to send text nobody needed.
• Different users may see different documents. Retrieval is where you enforce that, and a prompt is not.
• Content changes daily. Re-embedding one changed document is cheaper than re-sending everything.
• You need to show sources. Retrieval hands you the citation; a giant prompt makes you ask the model to remember where something came from, which it will occasionally invent.

The cost curve nobody projects

Stuffing the context is cheap at ten queries a day and expensive at ten thousand. The break-even arrives quietly, usually a month after launch when usage grows. Work out the cost per request at your expected volume — not your pilot volume — before choosing.

The middle path

Most systems we build end up hybrid: coarse retrieval to narrow thousands of documents down to a handful, then a generous context window so the model sees each of those in full rather than in fragments. This avoids the classic failure of early retrieval systems, where a chunk boundary sliced a table in half and the answer was confidently wrong.

The part that actually decides it

If the problem is knowledge — the model needs facts it does not have — retrieval or a bigger prompt both solve it, and you pick on cost and permissions. If the problem is behaviour — the model knows the facts but will not follow your format, tone or process — neither helps. That is a prompting, tooling or fine-tuning problem, and reaching for retrieval will waste a month.