Choosing between a large model and a small one
Model choice should follow the task, not the org chart. Extraction, classification and routing run well on small fast models; open-ended reasoning and long synthesis justify a larger one. Routing per task typically cuts both cost and latency substantially without a quality change users notice.
The default is to pick the best model available and send everything to it. It is a reasonable place to start and an expensive place to stay, because a large fraction of production traffic is doing work a much smaller model handles identically.
What small models are already good at
Classifying a message. Pulling four fields out of a form. Deciding which handler a question belongs to. Rewriting a sentence. Tagging content.
These share a shape: constrained input, constrained output, no real ambiguity. The gap between a small model and a large one on this kind of work is often too small to see, while the difference in cost and latency is large enough to feel.
Latency is a feature
A small model answering in a few hundred milliseconds makes interactions possible that a two-second response does not. Inline suggestions, live validation, anything that happens while a user is still typing. Sometimes speed is the product decision.
Where a large model earns it
Ambiguity, synthesis across many sources, multi-step reasoning, long documents, code that has to be correct, and anything where a subtly wrong answer is expensive. Here the difference is not marginal and trying to save money is a false economy.
Route, do not choose
The framing that helps is to stop asking "which model should we use" and start asking "which model should this call use". A summarisation endpoint and an autocomplete endpoint have no reason to share an answer.
A common and effective pattern is escalation: try the small model, check the result against a cheap criterion, and retry with the large one only when the check fails. Most traffic never escalates.
Measure on your data
Public benchmarks tell you about public benchmarks. Your task has its own distribution, its own edge cases and its own definition of good enough.
Assemble a hundred real examples with answers you trust, run every candidate against them, and look at the failures rather than the average. An afternoon of this replaces a month of opinion, and it gives you a harness that keeps paying off the next time you consider switching.
Frequently asked questions
Is it worth the complexity of running two models?
Usually yes, and the complexity is smaller than it sounds — one interface, a routing rule, two configurations. The saving on high-volume simple calls is often large enough to fund the rest of the feature.
How do I know a smaller model is good enough?
Build a set of a hundred or so real cases with known-good answers, run both, and compare. This takes an afternoon and replaces an argument that would otherwise run for weeks.
