Retrieval-augmented generation: architecture that holds up in production
A working retrieval pipeline is straightforward. Making it reliable enough to trust with business questions is where the engineering is.
ZooBieDoo Infotech Engineering TeamRead
Key takeaways
- Assess on genuinely messy documents, not clean templates.
- Set confidence thresholds per field by cost of error.
- The review workspace determines the system's throughput.
- Validate against systems of record before writing anything.
Real documents are not clean templates
Commercial extraction tools perform well on consistent forms and poorly on the photographed, handwritten and scanned material that makes up most genuine intake — which is exactly the part consuming the time.
Assess against a stratified sample weighted towards the difficult end, not against the tidy examples.
Confidence thresholds per field
The cost of being wrong differs by field. A reference number read incorrectly is an inconvenience; a payment amount read incorrectly is a serious problem.
Set thresholds per field according to that cost. Anything below routes to human review; anything above is written automatically. This single decision determines both the automation rate and the risk profile.
Design the review workspace
- Show the extracted value beside the region of the source it came from.
- Allow correction in one action, without re-keying the rest.
- Reject any extraction that cannot be traced to a source region.
- Capture corrections as labelled data for future evaluation.
The review interface determines the throughput of the whole system, and it is routinely built last with the least attention.
Validate against systems of record
Never write extracted values straight into a business system. Validate against what you already know: does the reference exist, does the amount fall within the policy limit, does the date make sense.
Business-rule validation catches a substantial share of extraction errors regardless of model quality, and it is cheap to implement.
Measuring the business outcome
Extraction accuracy is an engineering measure. The business measure is what proportion of documents complete without human involvement, and how much handling time fell.
Track straight-through processing rate, time per document end to end, and the rework rate where an automated extraction later proved wrong. These are what justify the investment and what reveal whether thresholds are set sensibly.
A system with high accuracy and low straight-through rate usually has thresholds set too conservatively, which is a tuning decision rather than a model problem.
Frequently asked questions
It depends heavily on document quality and field type. Printed structured fields extract very reliably; handwriting and poor scans considerably less so. Measure on your own documents before committing to a target.
Often not initially. General document-understanding services plus business-rule validation handles many cases. Custom training is worth it for high volume in a narrow, stable format.
Confirm language coverage in your extraction service and evaluate separately per language. Accuracy commonly varies more between languages than teams expect.
