Retrieval-augmented generation: architecture that holds up in production
A working retrieval pipeline is straightforward. Making it reliable enough to trust with business questions is where the engineering is.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Version prompts and deploy them through the pipeline.
- Explicit structure outperforms elaborate wording.
- Validate output against a schema with defined failure behaviour.
- Contain injection by limiting capability, not by instruction.
Treat prompts as versioned code
A prompt edited directly in a production console with no history is an untracked change to system behaviour. Store prompts in version control, review changes, and deploy them through the same pipeline as everything else.
Record which prompt version produced which output. Without that, debugging a bad answer from last week is guesswork.
Structure beats persuasion
Clear structure — explicit role, task, constraints, output format, examples — outperforms elaborate phrasing. Models follow well-specified instructions more reliably than they respond to emphasis.
- State what to do when the answer is not available.
- Specify output format precisely and validate it on receipt.
- Include a small number of representative examples, including an edge case.
- Keep instructions and data clearly separated.
Validate every output
Never trust output shape. Parse and validate against a schema, and define behaviour for failure: retry once with the error, then fall back or escalate.
Where the platform supports constrained or structured output, use it — it removes a whole class of parsing failure.
Guard against injection
When user content or retrieved documents enter the prompt, they can carry instructions. Keep a clear boundary between instructions and data, do not grant the model authority to take consequential actions on its own, and validate anything it produces that will be executed or displayed.
The reliable control is limiting what the system can do, not asking the model politely to ignore instructions in the data.
Managing cost and latency in the prompt
Prompt design has direct cost and latency consequences that are easy to overlook while iterating on quality.
- Long static instructions repeat on every call — trim them once behaviour is stable.
- Excessive examples add tokens for diminishing returns; test how few are actually needed.
- Requesting verbose reasoning increases both cost and response time.
- Where the platform supports caching of a shared prefix, structure the prompt to exploit it.
Measure tokens per request alongside quality in the evaluation run, so the trade-off is visible while the prompt is being tuned rather than afterwards.
Frequently asked questions
Against the evaluation set, in the pipeline, like any other change. Manual spot-checking finds obvious breakage and misses regressions.
Editable by non-engineers is convenient and risky. If you allow it, version and evaluate the changes the same way.
Only up to a point. Excessive context increases cost and latency and can dilute attention. Concise, well-structured instructions usually outperform long ones.
