01
Short answer
Fine-tune a small language model when the task is narrow, stable, high-volume, and supported by several hundred reviewed examples. Keep prompting a larger model while the workflow or expected output is still changing.
The default architecture for an LLM feature in 2026 is a large hosted model behind a carefully written prompt. That is correct for the first version. The prompt is where you should spend your early effort, because it is cheap to change and it teaches you what the task actually is.
But at some point the bill and the latency start to matter, and someone asks whether a smaller model could do the job. Once cost and latency matter, a fine-tuned small language model, an SLM, can outperform a prompted frontier model on three dimensions: cost per call, latency, and consistency of output format. It gives up breadth: the moment the task wanders outside its training distribution, a small model has nowhere to go, while a large model improvises.
The craft is knowing which situation you are in.
02
The decision, in one paragraph
Fine-tune an SLM when the task is narrow, you can write down what "correct" looks like, and you have at least a few hundred examples of input and desired output. Keep prompting the frontier model when the task is open-ended, the inputs are unpredictable, or you are still discovering what users actually ask. Most teams that regret fine-tuning jumped before the task was stable, and the model they trained encoded a workflow that changed three weeks later.
There is a middle path that too few teams take seriously: keep the frontier model, and make it cheaper through routing and caching, before committing to training anything. Our guide on LLM inference cost covers that side. This article is about what happens after those levers are exhausted.
03
What fine-tuning actually changes
A prompt is context you pay for on every request. Fine-tuning moves the instructions, the format, and much of the task knowledge into the weights, where you pay for them once. The practical consequences are bigger than the token arithmetic suggests.
A fine-tuned 8B-class model will follow your output format with a reliability that no prompted model matches, because format adherence is exactly what supervised fine-tuning teaches best. JSON schemas stop needing repair passes. Classification stops needing few-shot examples in the prompt. The prompt shrinks from thousands of tokens to hundreds, which cuts both cost and the latency of processing all that context.
What fine-tuning does not do is add knowledge the model has never seen in a form it can retrieve reliably. For factual grounding, retrieval still does the heavy lifting; our article on RAG grounding covers that division of labor. Fine-tuning shapes behavior. Retrieval supplies facts. Teams that try to use fine-tuning as a memory system end up retraining every time a fact changes, which is the most expensive possible database.
04
What the data needs to look like
The examples are the product. Everything else, the base model choice, the training setup, the hyperparameters, matters less than whether the examples are good.
A few hundred clean input-output pairs, drawn from real traffic, beat tens of thousands of scraped ones. If your task is extraction or classification, label real documents, and have two people label a sample independently so you know the agreement rate. If annotators agree with each other only 80 percent of the time, the model will learn your disagreement, not your intent. We wrote about judgment agreement in the AI evaluation guide, and it applies here with force, because a fine-tune is just evaluation in reverse: the same gold data, used for teaching instead of testing.
Hold out a test set before you train anything, from the same distribution as production traffic, and never let it leak into training. The first time teams skip this, they get a model that looks excellent and collapses in production, because they evaluated on their training examples and mistook memorization for capability.
05
The costs people forget
Training the model is the cheap part. An SLM fine-tune on a few thousand examples costs very little on any hosted tuning service. The real costs sit on either side of it.
Before training: the labeling effort, and the discipline to freeze the task definition. After training: the operational surface you just took on. A fine-tuned model is now your artifact to version, evaluate, and retrain when the upstream format changes or the business rules shift. The prompted frontier model updated itself last night whether you liked it or not; your SLM updates when you schedule it, and the drift between your frozen weights and the moving world is now your maintenance item.
This is why the evaluation harness has to exist before the fine-tune does. Retraining without a regression suite is how a format fix in March quietly destroys an extraction behavior nobody had a test for.
06
Where small models shine in production
The workloads where SLMs consistently win are the ones with high volume and narrow scope: document classification and routing, field extraction over a known document family, intent detection for support triage, response drafting inside a fixed template, and embedding-adjacent tasks where the output space is small and enumerable.
At thousands of calls a day, the cost difference between a fine-tuned small model serving on dedicated capacity and a frontier API compounds into a serious number, and the latency difference shows up in every user interaction. The Cloud Run GPU guide on this site works through the serving economics on the infrastructure side.
The pattern that works in delivery is a tiered one: the small model handles the traffic it is confident about, an escalation path routes the uncertain remainder to the frontier model, and the router is measured on how often it escalates. You get small-model economics with large-model safety, and the escalation rate becomes a dial you tune as the small model improves.
07
A sequencing that works
Start with the frontier model and a serious prompt. Build the evaluation set from real traffic. Add caching and routing, and measure what is left. If the residual workload is narrow, stable, and high-volume, collect examples from it, hold out a test set, fine-tune, and compare against the frontier baseline on the same evaluation. Ship the fine-tune only if it wins on that comparison, and keep the frontier path warm as the fallback.
Teams that follow this sequence rarely regret the fine-tune, because by the time they train, the model is the easy part.
08
What should trigger retraining?
Retrain when the task has changed enough that the current model fails a named slice of the evaluation set. New document formats, policy labels, languages, or output fields are valid triggers. A calendar date by itself is not.
Collect corrected production cases, remove duplicates, and separate changes in the task from ordinary model errors. If the prompt, retrieval source, or validation rule can fix the problem safely, use that smaller change. Training is justified when repeated behavior needs to move into the model and the examples describe the new behavior consistently.
Every training run gets a new held-out set containing old and new cases. Compare it with the serving model on quality, latency, memory, and cost. Release only when the new model improves the affected slice without breaking the stable ones.
Keep the prior model available through the observation window. Monitor disagreement on live traffic and route uncertain cases through the existing fallback. A rollback should switch the model version, not require a fresh deployment or emergency training run.
09
Related
Where to read next on the same subject.