Retrieval-augmented generation: architecture that holds up in production
A working retrieval pipeline is straightforward. Making it reliable enough to trust with business questions is where the engineering is.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Grade a real sample with a domain expert before committing.
- Field definitions and when they changed matter more than volume.
- Labelling needs guidelines, agreement measurement and its own budget.
- Fix quality in the pipeline or improvements decay.
Assess honestly before committing
Sample the data and have someone who knows the domain grade it. Completeness, accuracy, consistency of definitions across systems, and how far back reliable history goes.
This assessment frequently changes the project. Discovering during a proof of value that a critical field has been optional for two years is far cheaper than discovering it later.
Definitions matter more than volume
The same field often means different things in different systems, and the meaning changes over time as processes change. A status code that was repurposed three years ago will quietly corrupt any model trained across that boundary.
Document the meaning of each field, and when it changed. This is unglamorous work and it determines whether the results are trustworthy.
Labelling is a project of its own
- Write guidelines with worked examples, including hard cases.
- Measure agreement between labellers — low agreement means the task is underspecified.
- Have domain experts label a gold set for evaluation.
- Budget for it explicitly rather than absorbing it.
Fix the pipeline, not the snapshot
Cleaning an extract once produces a model that degrades as new data arrives with the same problems. Fix quality at the source or in the ingestion pipeline so improvements persist.
Add validation at ingestion that rejects or flags records failing basic expectations. Catching problems at entry is far cheaper than reconciling them later.
Monitoring quality after deployment
Data quality is treated as a preparation task and then stops being watched, which is when models degrade silently as upstream systems change.
Add automated checks on incoming data: expected value ranges, null rates, category cardinality and record volumes. Alert when they move beyond normal variation, because that is usually the earliest signal of a model problem.
These checks catch upstream changes — a renamed category, a changed unit, a broken integration — long before accuracy metrics reflect them.
Frequently asked questions
It depends on the task. Retrieval and extraction can work with modest corpora; learning a subtle pattern from scratch needs considerably more. Quality and representativeness matter more than raw volume.
For augmenting rare cases and for testing, yes. As a substitute for real data it tends to encode the assumptions used to generate it.
The team that owns the source system, with support. Making it the modelling team's responsibility moves the fix far from the cause.
