A working retrieval pipeline is straightforward. Making it reliable enough to trust with business questions is where the engineering is.
A retrieval system chunks documents, embeds them, stores the vectors, retrieves candidates for a query, reranks them and passes the best to a language model with instructions to answer only from what it was given.
Each stage leaks quality. Chunks that split a table from its caption lose meaning. Embeddings that do not match the domain retrieve plausible but wrong passages. A model given weak context will still produce a confident answer.
Fixed-size chunking is simple and frequently wrong, cutting mid-sentence and separating headings from content. Structure-aware chunking — splitting on document sections and keeping headings with their body — is usually a larger improvement than changing the model.
Retain metadata with each chunk: source document, section, effective date and access permissions. Without it you cannot cite properly or filter reliably.
Require citation to a specific passage, not a document. If the system cannot point at supporting text, it should decline rather than answer. This constraint removes most fabricated detail because it makes fabrication structurally difficult.
Build an explicit refusal path. A model with no better option will attempt an answer; give it one, and users will trust the answers it does give.
Two production concerns are routinely deferred and both are difficult to retrofit. The first is access control: retrieval must filter by the asking user's permissions before ranking, not after, or the system will surface content the user should not see.
The second is freshness. Documents change and are superseded, and a corpus without an update pipeline drifts steadily out of date while continuing to answer confidently.
Build both into the ingestion design: carry permissions and effective dates as metadata on every chunk, and re-index on source change rather than on a schedule that guesses.
For modest corpora, a vector extension on your existing database avoids operating another system. Dedicated stores earn their place at larger scale or where advanced filtering and hybrid search matter.
Large enough to carry meaning, small enough to be precise often a few hundred tokens with overlap. Test against your own evaluation set rather than adopting a default.
Rarely for knowledge tasks. Retrieval supplies the knowledge; fine-tuning is more useful for output format, tone or a narrow classification task.
Keep reading
Connecting a language model to your documents is the straightforward part. The reliability work is everything after that, and it looks nothing like the demonstration.
Without an evaluation set, every change to a language model system is argued on anecdote. Building one is the highest-value early task.
Governance is largely about being able to explain what a system does, on what data, with what oversight, and to evidence it.
