Retrieval-augmented generation: architecture that holds up in production
A working retrieval pipeline is straightforward. Making it reliable enough to trust with business questions is where the engineering is.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Build the evaluation set before the pipeline, not after.
- Include unanswerable questions — the most commonly missing category.
- Measure retrieval separately from generation.
- Track cost per query alongside quality on every change.
Build the set before the pipeline
A few hundred questions with verified answers, drawn from real user queries where possible, is enough to start. It must cover four categories deliberately.
Answerable questions the corpus supports. Unanswerable ones where the correct behaviour is to decline — usually missing and the most important. Ambiguous ones where the system should ask. Adversarial ones with near-miss retrievals and superseded documents.
Choose measures that match the task
- Exact or fuzzy match for extraction and classification.
- Groundedness — is every claim supported by retrieved context?
- Retrieval hit rate at k, measured independently of generation.
- Refusal correctness — declining when it should, answering when it should.
Model-graded evaluation, used carefully
Using a language model to grade outputs scales where human review cannot. It is useful and it is not ground truth: graders have biases, favour verbose answers and can be inconsistent.
Calibrate the grader against human judgements on a sample, and re-check periodically. Treat model grading as a screening tool with human review on a rotating subset.
Run it on every change
Wire evaluation into the pipeline so prompt edits, model version changes, chunking changes and threshold adjustments are all measured. Without this, improvements and regressions are indistinguishable.
Track cost per query alongside quality. An accuracy gain that triples inference cost is a trade-off someone should make deliberately rather than discover on an invoice.
Turning production failures into tests
The evaluation set should grow from real failures. When a user reports a wrong answer, add that question with the correct answer to the set before fixing anything.
This does two things: it prevents the same regression recurring, and it steadily makes the set representative of what users actually ask rather than what the team imagined.
Review the additions periodically to keep the set balanced. A set dominated by one failure mode will over-optimise for it, which is its own kind of drift.
Frequently asked questions
A few hundred well-chosen items is enough to detect meaningful regressions. Breadth of scenario matters more than raw count.
They are useful for comparing base models and tell you little about your corpus and your users' questions. Your own set is what matters for your system.
Whenever production surfaces a failure the set did not catch. Adding those cases is how the set stays representative.
