When a small model is the right choice
One afternoon I pointed three pipelines at a model a fraction of the size of the one they had been running on. Same prompts, nothing else touched.
Two carried on as though nothing had happened. The third fell over inside a minute.
What I wrote down that evening was that the small model could not handle real work. I held that view for about a week. It was wrong, and it was wrong in an expensive direction, because it takes a working tool off the table for the large majority of steps where it had been completely fine.
The pipeline that broke reads fifteen minutes of transcript and decides what the whole thing was about. The two that shrugged are a paragraph tagger and a mail classifier. Nobody needed an experiment to predict that split. I had simply never asked the question, because all three ran on one model for the reason most systems end up that way. It was already there and nothing forced a decision.
None of what follows is about self hosting or licences. A small model can be an endpoint somebody else runs. The axis is capability against what the task actually demands, and that stays live whatever you decide about where the thing lives.
Where the size stops mattering
There is a category of work where I have looked hard and cannot tell the two apart.
Classification into a fixed set of labels. Five options, a clear description of each, and the answer is one of the five.
Extraction where the field is already sitting in the text. The amount and the reference out of a payment notification. The number is right there. Nothing is being worked out.
Routing a request to one of the handlers you wrote.
Reformatting. Take this, give it back in that shape.
Short mechanical rewriting. Under a hundred characters. Drop the greeting. Turn the statement into a question.
The common property is that the answer already exists inside the input and the model’s job is to locate it and put it somewhere. No chain of inference. No holding two distant things together. No judgement a careful reader would want to argue with.
I tagged four thousand paragraphs both ways and spent an afternoon reading the disagreements by hand. Nearly all of them were cases where the correct answer was genuinely arguable and either label was defensible. That is the signature of a task that is flat with respect to model size, and if your step looks like one of these, the large model is doing work nobody asked it for.
Four weaknesses, said plainly
Reasoning across several steps. Anything shaped like work out A, use A to get B, check B against C. The small model returns something that reads like an answer with a step missing from the middle. It comes back fluent, which is what makes it dangerous. Fluent and wrong takes far longer to catch than confused and wrong.
Long context where the answer lives in two places at once. Big context windows on small models are normal now, and they will reliably find one relevant passage. Ask something that needs paragraph two and paragraph forty held together and the failure rate climbs sharply. It answers from whichever half it found first and sounds equally certain either way.
Instructions that cut against the grain. Ordinary instructions land fine. An unusual one, say never answer the question, only rewrite it as a sharper question, holds for a while and then quietly lapses. The model drifts back toward the behaviour it has seen most. I have watched that happen inside a single output: correct for two paragraphs, then back to answering.
Malformed input, which is the one people do not see coming.
On clean input a small model is fine. Hand it something outside what it expected, truncated, wrongly encoded, empty, in another language, a hundred kilobytes of HTML where a sentence should have been, and it degrades badly rather than gracefully. A larger model tends to notice that the input is broken and say so. The small one carries on and produces a confident, well formed answer about nothing.
My mail step ran for six days before one arrived: a forward nested four levels deep with the encoding mangled somewhere in transit. The step returns one word out of five. It returned two sentences apologising for not being able to decide. Nothing downstream had ever been written to handle a sentence, so the message went to a folder I look at once a fortnight and sat there.
The failure mode is the part worth internalising. It did not error. It did not stall. It produced the usual shape of output at the usual speed and only the contents were rubbish, so every alarm I had built around latency and error rates stayed green. That asymmetry will also never show up in your evaluation, because you build the test set from representative inputs and these arrive from outside the set entirely.
The arrangement that survives
Both models, with a seam between them.
The small model takes the volume. The larger one sits behind it and receives whatever the small one flags or fails on. My mail step works that way now and the large model wakes for somewhere around five percent of traffic.
All the design is in the trigger, and it deserves more thought than the model choice did.
Start with a validator. The answer must be one of your five labels, or match a shape you can check in code. If it does not, escalate. That single check catches the apologising sentence, and it catches it in the first hour instead of the second week.
Then give the small model permission to give up. Add a label meaning I do not know, describe exactly when to reach for it, and route that one upward. Models flag their own uncertainty far better when you have given them somewhere to put it than when refusal is off the menu.
Then guard the input before you spend a call on it. Length outside a sane range, an encoding that fails to decode, an empty body, a ratio of markup to words that says web page rather than message. Ten lines each, and they catch precisely the class of failure the small model handles worst.
Then sample. Take one in fifty of the answers that passed the validator, run them past the large model anyway, and log the disagreements. That is how you find out your escalation rate is wrong before something teaches you the hard way.
The version I built badly
My first trigger escalated about a third of the traffic.
Every escalated item pays the latency of both calls back to back. The median got slower, the tail got considerably worse, and I had built the poorer half of each option. It took a week of staring at the wrong graph before I understood what I was looking at.
Too loose is the default failure. A validator that fires on formatting noise rather than on wrong answers will send you perfectly good work upstairs all day, and nothing in your monitoring reads that as a problem.
Why speed and control beat the invoice
People reach for a smaller model to save money, then talk about the money, and the two better arguments get buried underneath it.
Speed first. A small model starts producing sooner and finishes sooner, and the gap is wide enough to feel. If a person is sitting in front of the thing waiting, that is worth more than any line on a bill.
Control is the one I underrated. Small is cheap enough to run over everything instead of a sample. Cheap enough to run twice on the same input and compare the two answers, which is a real quality mechanism you cannot afford at the top of the price list. Cheap enough to retry hard rather than design around a retry budget. That changes what you are willing to build. I check every paragraph now instead of spot checking, because checking everything stopped being a decision anyone had to make.
The mistake underneath the whole thing
Swapping the big model out everywhere at once is how most people arrive at the belief that small models do not work in production.
The failure there is architectural. I had one model doing five different jobs because that is how the system grew, and I replaced the model when the actual problem was that the jobs had never been separated. Once the transcript summary went back up top and everything else stayed down, both halves worked. Nothing about the model changed between those two states.
Reusing the same prompt is the other half. A prompt tuned against a large model leans on that model filling in gaps. A small one wants the edge cases spelled out, the output format stated flatly, and one example of exactly what you want. That is an hour per step, and skipping it makes the comparison meaningless. Fine tuning sits one rung further along and is a different conversation.
I cannot tell you where your own line falls. I have four steps running on small models and I got to those four by testing, not by reading, and the honest version of that test is two hundred real inputs you already know the answers to, both models, and an afternoon reading the disagreements yourself.
There is also a gap in what I know. I do not run small models inside tool calling loops of any depth, so when somebody asks whether one can drive an agent through fifteen steps, I have no answer. The reasoning weakness suggests probably not. I have not proved it, and I would rather say so than guess confidently. The comparison notes I keep on each step I have moved down are here.