Building a retrieval demo makes for a good weekend. You pick an embedding model, push your documents into a vector store, point an LLM at the top results, and by Sunday night the thing answers questions about your own data. It feels finished.
Then real people start using it, and the gaps show up fast.
It quotes a document that does not actually contain the answer. It makes up a number when the right page never got retrieved. Someone in finance asks the same question two slightly different ways and gets two different answers. Response time drifts from one second to six during busy hours. The bill from your model provider doubles in a month and nobody can explain why.
None of that is a model problem. It is the distance between a demo and a system you can run. The model is maybe a fifth of the work. The rest is the unglamorous machinery around it, and that machinery is what decides whether the project survives its first month in front of users.
Here is the shape of a RAG system that holds up.
1. Retrieval is the part that actually breaks
Almost every bad answer we trace back ends in the same place: the model never saw the right text. People reach for a bigger or smarter model when the real fix is upstream, in how documents get split and found.
Chunking sounds trivial and is not. Cut documents too small and you lose context. Cut them too big and the embedding turns into mush that matches everything and nothing. Tables, headings, and code blocks each want different handling. We usually start simple, look at what comes back for real questions, and adjust from there rather than guessing up front.
Then there is the question of whether plain vector search is enough. Often it is not. Adding keyword search alongside it, or a reranking step that re-scores the top twenty results, fixes more problems than swapping the model ever will.
2. If you cannot measure quality, you are guessing
This is the step teams skip, and skipping it is why they cannot tell whether last week’s change helped or hurt.
Before we ship anything, we build a small set of real questions with known good answers. A few dozen is enough to start. Every change to chunking, retrieval, or the prompt runs against that set, and we score two things separately: did retrieval bring back the right source, and did the answer use it correctly. Keeping those scores apart matters, because a wrong answer from the right document is a prompt problem, while a wrong answer from the wrong document is a retrieval problem. They need different fixes.
Once that loop exists, you stop arguing about whether the system feels better and start seeing whether it is.
3. Guardrails for when the answer is not there
A confident wrong answer is worse than no answer. The most useful thing we add is the ability to say “I do not have that information,” and to mean it.
That means detecting when retrieval came back weak and refusing to dress it up. It means staying inside scope, so a support assistant does not start giving legal opinions. And it means a fallback path for when the model or a tool times out, because both will, usually at the worst moment.
4. Cost and latency are features
Treat them as afterthoughts and they will quietly decide whether anyone keeps using the thing.
A few habits keep both under control. Cache answers to repeated questions, because the same handful of questions tends to make up a surprising share of traffic. Use a small, cheap model for routing and a larger one only when the work needs it. Set a token budget per request and stick to it. Put a hard ceiling on retrieved context so one giant document cannot blow up a single call.
We attach a rough cost-per-request number early and watch how it moves as usage grows. If it climbs in the wrong direction, better to know in week two than in the next invoice.
5. Monitoring you would actually open
Log every query, the documents it retrieved, and the answer it gave. That single habit turns “users say it is wrong sometimes” into a list of specific cases you can reproduce and fix.
On top of that, track latency, cost, and some signal of whether answers are landing, even something as basic as a thumbs up or down. Then set a couple of alerts that would wake the right person if quality or spend fell off a cliff. Not a wall of dashboards nobody reads. Two or three numbers that matter.
6. A plan for keeping the index fresh
Your documents change. New policies, new products, edited pages. If the index does not change with them, the system starts confidently quoting last quarter’s answer, and that erodes trust quickly.
So there has to be a process for re-indexing when content updates, ideally automatic, and a way to remove documents that should no longer be searchable. It is not hard. It just has to exist before launch, not after someone notices the assistant recommending a discontinued plan.
Where teams usually get stuck
The pattern is almost always the same. The prototype works in the room. It demos well to the boss. Then it meets a hundred real users with messy questions and edge cases nobody scripted, and the team realises the hard part was never the model. It was everything in that bottom band of the diagram.
That is a normal place to be stuck, and it is fixable. Most of the work is steady and unspectacular, which is exactly why it gets left out of demos.
If you have a RAG prototype that works in a controlled setting but not in front of users, this is the kind of work we do day to day. You can read more about our approach to AI engineering, or book a scoping call and we will help you find the gap.