Chapter 23: When to Fine-Tune vs. When to Prompt

The Fine-Tuning Temptation

When prompting does not produce quite the right output, fine-tuning feels like the obvious solution. 'If I train the model on examples of what I want, it will learn to produce it reliably.'

Sometimes this is correct. Often it is the wrong solution, chosen because it feels more technically substantial than prompt engineering.

This chapter gives you the framework to make the right choice.

---

What Fine-Tuning Actually Does

Fine-tuning updates the model's weights on a dataset of (input, output) pairs. It effectively teaches the model new patterns by adjusting its parameters.

Fine-tuning is powerful for:

• Teaching the model a completely new style that prompting cannot reliably produce
• Teaching the model to follow a very specific output format consistently
• Teaching the model domain-specific knowledge not present in its training data
• Making a task more efficient (fine-tuned models often need shorter prompts, reducing token costs)

Fine-tuning is NOT powerful for:

• Teaching the model new facts (it will often hallucinate from memory even after fine-tuning)
• Tasks that prompting already handles well
• Rapidly evolving tasks where your training data goes stale quickly

---

The Decision Framework

Question 1: Have you exhausted prompt engineering?

Before considering fine-tuning, ensure you have tried:

• Detailed role prompt with domain expertise
• Clear output format specification with examples
• Few-shot examples (5-10 examples in the prompt)
• Chain-of-thought reasoning
• Tested at least 3 different model families

If any of these techniques have not been tried, try them first.

Question 2: Do you have enough high-quality training data?

Fine-tuning requires:

• OpenAI GPT-4o fine-tuning: Minimum 10 examples, but 50-200+ for quality results
• True quality results: 500-2000 carefully curated examples
• Consistent quality: Every training example must be correct

If you cannot produce 200+ high-quality (input, output) pairs, fine-tuning is premature.

Question 3: Is the task style-dependent or knowledge-dependent?

Style tasks (fine-tuning often wins):

• Writing in a very specific brand voice that prompting cannot reliably replicate
• Following an unusual format that requires many tokens to specify in every prompt
• Producing output that matches a proprietary schema exactly

Knowledge tasks (prompting + RAG usually wins):

• Answering questions about your product's documentation
• Responding based on your company's policies
• Using information that updates regularly

For knowledge tasks, fine-tuning the knowledge in is almost always worse than using RAG. The model's knowledge cutoff will make the fine-tuning stale.

Question 4: What are the ongoing costs?

Fine-tuning has two cost components:

• Training cost: One-time cost to train the model
• Inference cost: Fine-tuned models are often priced differently from base models

OpenAI fine-tuning costs (approximate, 2026):

• Training: $0.008 per 1K training tokens
• GPT-4o fine-tuned inference: 2-3x base model pricing

---

When Fine-Tuning Clearly Wins

Scenario 1: Consistent format adherence Your application generates structured reports in a complex proprietary format. Despite detailed format instructions and examples in the prompt, the model occasionally deviates. Fine-tuning on 500 correct format examples produces near-100% format compliance.

Scenario 2: Style replication You are building a tool that writes in a specific author's style. No amount of prompting produces truly consistent style replication. Fine-tuning on 1,000 examples of the author's work produces dramatically better results.

Scenario 3: Cost optimisation at scale Your application makes 50M model calls per month. Your current prompt is 800 tokens. Fine-tuning on your specific task allows you to reduce the prompt to 200 tokens while maintaining quality. The token savings at 50M calls per month significantly outweigh the fine-tuning cost.

---

Practical Guidance

Start with this hierarchy:

• Zero-shot prompt
• Few-shot prompt (5-10 examples)
• Optimised system prompt + few-shot
• RAG (for knowledge-dependent tasks)
• Fine-tuning (only when 1-4 have been exhausted and requirements justify the investment)

Fine-tuning is a tool. It is not the solution to every prompt engineering challenge.