Retrieval-augmented generation: architecture that holds up in production
A working retrieval pipeline is straightforward. Making it reliable enough to trust with business questions is where the engineering is.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Version data, code, parameters and artefact together.
- Separate training from serving; they have opposite requirements.
- Training-serving skew is a common silent failure.
- Trigger retraining on measured drift, not on the calendar.
Reproducibility first
A model you cannot rebuild is a liability. Version the training data, the code, the parameters and the resulting artefact together, so any model in production can be traced to exactly what produced it.
Data versioning is the part usually skipped and the part that matters most when a model behaves unexpectedly six months later.
Separate training from serving
Training is batch, tolerant of latency and resource-hungry. Serving is latency-sensitive and needs to be reliable. Conflating them produces a system that is bad at both.
Package the model behind a stable interface so the serving layer does not change when the model does.
Watch for the two kinds of drift
- Data drift — inputs change distribution; the model is now extrapolating.
- Concept drift — the relationship between inputs and outcome changes.
- Training-serving skew — features computed differently in the two paths, a very common silent failure.
Monitor input distributions and prediction distributions, not only errors. Errors often arrive too late to be useful.
Retrain on evidence
Retraining on a fixed calendar wastes effort when nothing has changed and is too slow when something has. Trigger on measured degradation or drift instead.
Automate the retraining path and keep a human approval before promotion, with the evaluation comparison in front of them. A model that retrains and deploys itself unattended will eventually deploy something worse.
Deploying models safely
Model releases deserve the same caution as code releases and often get less. Shadow deployment — running the new model alongside the current one without acting on its output — reveals behaviour on real traffic with no risk.
Follow with a gradual rollout, comparing business outcomes rather than only offline metrics. A model with better accuracy that shifts the distribution of decisions can produce worse business results.
Keep the previous model available for immediate rollback, and treat a model rollback as a rehearsed operation rather than an improvisation.
Frequently asked questions
For one or two models, a well-structured pipeline with existing CI tooling is usually enough. Platforms earn their cost with many models and several teams.
When monitoring indicates degradation. Some domains drift weekly, others are stable for years. The measurement matters more than the schedule.
They solve training-serving skew by computing features once and serving them to both paths. Valuable when features are shared across models; overhead when they are not.
