Chapter 9: Prompt Debugging — Diagnosing and Fixing Failing Prompts

The Random Edit Trap

Your prompt is not working. The output is wrong, incomplete, off-format, or just bad. What do you do?

Most people make a random change — add a sentence, rephrase a section — run it again, see it is still wrong (or now wrong in a different way), and repeat. After 15 minutes of random edits, the prompt is worse than it started.

Prompt debugging requires the same systematic approach as code debugging: hypothesise the failure mode, isolate the variable, test the fix.

---

The Five Failure Modes

Almost every prompt failure falls into one of five categories:

Failure Mode 1: Ambiguity

Diagnosis: Ask the model: 'Before you answer, tell me how you are interpreting this request.' Compare its interpretation to your intent.

Example:

• Prompt: 'Write a short email to the customer about their order.'
• Model's interpretation: A casual, brief note.
• Your intent: A formal status update with specific order details.
• Fix: Specify 'Write a formal status update email (2-3 paragraphs). Include: order number, current status, expected delivery date. Tone: professional and apologetic for any delays.'

Failure Mode 2: Missing Context

Diagnosis: Read the output and ask: 'What information did the model assume that it should have been given?'

Example:

• Prompt: 'Write a LinkedIn post about my product launch.'
• Problem: Model invents generic product details.
• Fix: Provide the product name, what it does specifically, the target audience, the key differentiator, and the specific CTA you want.

Failure Mode 3: Instruction Conflict

Diagnosis: Read your prompt carefully and look for pairs of instructions that might conflict:

• 'Be concise' + 'Cover all these 8 points in detail'
• 'Write for beginners' + 'Assume familiarity with [technical concept]'
• 'Be definitive' + 'Present multiple perspectives'

Fix: Resolve the conflict explicitly. If you need both behaviors, specify the priority: 'Cover all 8 points, but use no more than 2 sentences per point.'

Failure Mode 4: Format Mismatch

Diagnosis: The content is right but the presentation is wrong.

Fix: Add an explicit output format specification with a skeleton/template.

Failure Mode 5: Task Complexity Overload

Diagnosis: Count how many distinct tasks your prompt asks for. If it is more than 3, complexity overload is likely.

Fix: Break the prompt into multiple sequential prompts (prompt chaining).

---

The Prompt Debugging Protocol

Follow these steps in order when a prompt fails:

Step 1: Run the prompt 3 times. Is the failure consistent? If it appears only 1 in 3 times, you have a sampling problem. If it is consistent, you have a structural problem.

Step 2: Ask the model to explain its interpretation. Prepend: 'Before answering, in one sentence, tell me how you understand this request.' The model's stated interpretation often immediately reveals the ambiguity.

Step 3: Strip the prompt to its core. Remove everything except the most essential instruction. Does the core task work? Gradually add elements back until you find what breaks it.

Step 4: Check for conflicts. Read every sentence in your prompt and ask: 'Could any other sentence contradict this?'

Step 5: Add an example. If you have not already included an example of desired output, add one. This single change fixes a surprising number of format and interpretation failures.

Step 6: Try a different model. Some failure modes are model-specific. If the problem disappears with a different model, the issue is a quirk of the original model's behavior.

---

The Prompt Variant Testing Method

When you suspect a specific part of your prompt is causing the failure, test variants:

const basePrompt = 'Analyse this customer feedback and extract the main themes.';
const variants = [
  basePrompt,
  basePrompt + ' Output as a numbered list.',
  basePrompt + ' Output as JSON: { themes: [{ name: string, frequency: "high"|"medium"|"low" }] }',
  basePrompt + ' First, think through all the themes you notice. Then list them in order of frequency.',
];

const results = await Promise.all(variants.map(v => callModel(v, feedbackText)));

Systematic variant testing is the fastest path to understanding which part of a prompt is driving which behavior.

---

Building a Failure Log

Every time a prompt fails in production, log:

• The exact prompt
• The failure type (from the 5 categories above)
• The fix applied
• Whether the fix worked