Managing the cost of AI systems in production

Know your unit cost

Track cost per request, per user and per resolved task from the first day. An aggregate monthly figure tells you the bill; the unit cost tells you whether the system is economic.

Log token counts alongside outcomes so the relationship between cost and value is visible rather than inferred.

Route by difficulty

Not every request needs the largest model. A smaller model handles routine cases well, with escalation to a larger one when confidence is low or the task is complex.

• Classify difficulty cheaply, then route.
• Cache responses for repeated identical queries.
• Use retrieval to keep context small rather than sending entire documents.
• Batch asynchronous work where the platform offers reduced rates.

Context length is the usual culprit

Sending large amounts of context on every request is the most common cause of unexpected cost. Better retrieval that sends five relevant passages instead of fifty reduces cost and usually improves quality.

Trim conversation history deliberately rather than accumulating it indefinitely.

Set limits before you need them

Rate limits per user, spending alerts and a hard cap prevent a loop or an unusual usage pattern turning into a large invoice.

Build these in at the start. They are simple then, and they are written under pressure after an incident otherwise.

Forecasting cost before committing

Cost surprises usually come from usage patterns nobody modelled. Before committing to a design, estimate tokens per request for each request type and multiply by realistic volume, including the retries and failures that will happen.

Model a pessimistic case as well: what if usage is five times the estimate, or if users write much longer inputs than expected? Knowing the ceiling informs whether limits are needed and where.

Revisit the model against actuals after a month in production. The gap between estimate and reality is usually instructive and worth feeding into the next design.