Fine-tuning vs prompting

There are two ways to make a general model behave the way your product needs. You can prompt it: give instructions, examples and context at the moment you call it. Or you can fine-tune it: train it further on your own examples so the behaviour is built in. Almost every successful agent starts with prompting, and many never need anything else. Knowing when that stops being true saves weeks of wasted effort.

Prompting#

Prompting means everything you control at request time: the system prompt, a few worked examples, the documents you retrieve, the tools you define and the output format you demand.

Its strengths are speed and flexibility. You can change behaviour in minutes, compare versions on the same test cases, and switch to a newer model the day it ships. Its costs are tokens (long instructions are resent on every call, though caching softens this) and the fact that instructions are followed most of the time, not always.

Before you decide prompting is not enough, check that you have used its full range:

• clear, specific instructions with the reasons behind the rules (writing instructions);
• two or three examples of ideal output, including a tricky case;
• a required structure for the answer (structured outputs);
• tools that give the model real data instead of asking it to remember;
• retrieval for facts that change or are private (RAG).

Fine-tuning#

Fine-tuning continues training a model on a set of example inputs and the outputs you want. It can make a model more consistent in a narrow task, teach a house style or format, improve accuracy on a specialised skill, or let a small cheap model imitate a bigger one for one job (often called distillation).

It has real costs:

• You need good data. Hosted services accept small sets (OpenAI's guide says the minimum is 10 examples and reports improvements from 50 to 100), but a robust result on a broad task usually takes more, often hundreds of reviewed examples. Bad examples teach bad habits.
• It is tied to a model version. When a better base model arrives you may need to repeat the work.
• It does not add reliable knowledge. Fine-tuning shapes behaviour and style; it is a poor way to teach facts, which are better retrieved at request time.
• It needs an evaluation set to prove it helped, which you should have built anyway (eval datasets).
• Not every provider or model offers it (OpenAI, for example, states that it is winding down its fine-tuning platform and no longer opens it to new users), and open-weight models can be tuned on your own hardware (local and open models).

A decision ladder#

What agents change#

An agent runs many model calls per task, so a small improvement in each call compounds. That makes consistency valuable, which is one reason teams do fine-tune for tool use or for a narrow classification step inside a larger agent. But the cheaper wins come first: tighter tools, shorter instructions, caching, and sending the easy steps to a small model.

Where to go next#

If the real problem is missing knowledge rather than behaviour, continue with what is retrieval-augmented generation.