Small models are quietly taking the boring jobs

Most of what production AI actually does is unglamorous: decide which of six categories this message belongs to, pull four fields out of a PDF, route a ticket, flag a document for review. None of that needs the largest available model, and running it on one is a habit worth breaking.

Why the default is oversized

Prototypes get built on whatever model is best, because at prototype volume the cost is invisible and you want to know whether the idea works at all. Then it ships, volume arrives, and nobody revisits the choice. The model that proved the concept quietly becomes the model that runs the business.

The jobs that move down a tier well

• Classification into a fixed set of categories, especially with a handful of examples in the prompt.
• Extraction of specific fields from a predictable document type.
• Routing and triage, where the output is one of a few known destinations.
• Reformatting — turning messy input into clean structured output.

What does not move down well: multi-step reasoning, ambiguous instructions, anything where the model has to decide what the task even is. Keep the capable model for the step that needs judgement, and use a cheaper one either side of it.

How to do it without guessing

This is exactly what an evaluation set is for. Run the same cases through the smaller model, compare against the outcomes you expect, and look at the failures rather than the headline score. Often the small model is within a point or two, and the errors are a category you can fix with a better prompt or a validation rule. Sometimes it falls apart and you have your answer in an afternoon.

The second benefit

Smaller models are faster, and latency changes what a feature can be. Under a second, a classification step can sit inside a user interaction. At five seconds it has to become a background job with a notification. Moving down a tier occasionally simplifies the whole architecture, which is worth more than the token saving.

Where this leaves the big models

Firmly in the picture, for the work that deserves them — planning a multi-step task, handling genuinely open-ended input, writing the first draft of something. The point is not that small models are better. It is that a production system usually contains several jobs of different difficulty, and pricing all of them at the hardest one is a decision most teams never consciously made.