The first invoice from a launched AI feature tends to arrive with a shock, and the first user complaint is usually about how long the spinner ran. Both problems trace back to the same habit: sending every request to the largest model with the whole conversation attached. There is a lot of slack to reclaim before you touch answer quality.
Route by difficulty instead of using one model for everything
Most traffic is easy. Classification, extraction and short rewrites do not need a frontier model, and a smaller one answers them faster and for a fraction of the price. Reserve the expensive model for the genuinely hard requests, and let a cheap classifier or a confidence threshold decide which is which.
- Send simple, high-volume tasks to a small fast model.
- Escalate to a larger model only when the cheap one is uncertain or the task is open-ended.
- Measure quality per route so a downgrade never ships silently.
Cache the parts that repeat
Prompt caching on a stable system prompt and shared context cuts both cost and time-to-first-token, because the provider skips reprocessing the prefix. Put the invariant material first and the variable user input last so the cacheable prefix is as long as possible. For deterministic tasks, an exact-match response cache keyed on the normalised input removes the model call entirely.
Stream, and stop paying for tokens you discard
Streaming the response changes perceived latency more than any backend optimisation — the user reads while the model writes. On the input side, trimming a bloated system prompt, summarising old conversation turns and capping retrieved context all reduce the tokens billed on every single call, which is where the real money leaks.
- Stream tokens to the client so the first word appears in well under a second.
- Set a sane max output length; unbounded generations are slow and expensive.
- Summarise or truncate conversation history instead of resending it verbatim forever.
Optimise against a benchmark, not a hunch
Every optimisation here trades some capability for speed or cost, and the only way to know whether the trade was acceptable is to re-run your evaluation set after each change. Track cost per request, p95 latency and quality together on one dashboard so a cheaper configuration that quietly got worse cannot hide.