Retrieval is not enough: what makes an enterprise AI assistant reliable

The gap between a demo and a system

A retrieval-augmented assistant can be assembled quickly and takes considerably longer to make trustworthy. The gap between those two states is where most enterprise AI programmes stall, and it is rarely closed by a better model.

The demonstration works because the questions asked of it are the questions it was built around. Production fails differently: users ask questions the corpus cannot answer, ask about documents superseded last quarter, ask two questions in one sentence, and — most damagingly — ask questions the system answers confidently and wrongly.

That last category ends programmes. A system that is usually right but cannot tell you when it is not transfers the verification burden to the user while removing the signal they would use to do it.

Build the evaluation set before the pipeline

Without an evaluation set, every subsequent decision is argued on anecdote. A workable starting point is a few hundred questions with verified answers, drawn from real user queries where possible and covering four categories deliberately:

• Answerable — the corpus contains the answer and it should be found.
• Unanswerable — it does not, and the correct behaviour is to say so. This category is usually missing and is the most important one.
• Ambiguous — the question needs clarification and the system should ask.
• Adversarial — near-miss retrievals, superseded documents, contradictory sources.

Every change to the system — chunking, embedding model, prompt, reranker, threshold — runs against this set. When a change feels better, the set tells you whether it is. A good share of changes that feel better are not.

Grounding, refusal and confidence

Three mechanisms do most of the reliability work, and none of them is glamorous.

Citation to source region. Not document-level citation but region-level. If the system cannot point at the passage supporting a claim, that claim does not ship. This constraint alone removes most fabricated detail, because it makes fabrication structurally difficult rather than merely discouraged.

An explicit refusal path. A model asked a question outside its corpus will attempt an answer unless the system offers it a better option. “I do not have a source for this — here is who does” is a valuable output, and users trust systems that produce it.

Thresholds tuned per intent. The cost of a wrong answer is not uniform. A policy lookup and a payment instruction warrant different thresholds; treating them the same means over-blocking the cheap case or under-blocking the expensive one.

Most failures are corpus failures

When an assistant answers badly the instinct is to change the prompt. A large share of production failures trace instead to the corpus: superseded documents that were never removed, several versions of the same policy with no recency signal, scanned files that extracted as noise, and access-controlled material the retriever could see but the asking user should not.

Corpus hygiene is unglamorous and it is the work. Ownership per document set, an expiry policy, extraction quality checks, and permission filtering applied at retrieval rather than at display. That last point is a security matter and it is got wrong regularly.