Chapter 18: A/B Testing Your Prompts in Production

Why Prompt Changes Feel Better Than They Are

You update a prompt. The next few outputs look better. You ship it.

Three weeks later, a user complains about a regression in output quality that you cannot reproduce. You roll back. The metrics are unclear. You do not know if the original or new prompt was actually better.

This is the default prompt management cycle. It is unreliable because it relies on:

• Anecdotal sampling (a few outputs is not a representative sample)
• Recency bias (new things feel better)
• No control group (you do not know what the old prompt would have produced on the same inputs)

---

The Prompt A/B Test Framework

Step 1: Define Your Success Metric

Before running any test, define what "better" means:

const successMetrics = {
  code_generation: {
    primary: 'test_pass_rate',
    secondary: ['compilation_rate', 'security_score'],
    guardrails: ['harmful_content_rate < 0.001'],
  },
  content_generation: {
    primary: 'human_preference_rate',
    secondary: ['readability_score', 'keyword_density'],
    guardrails: ['factual_accuracy > 0.95'],
  },
  classification: {
    primary: 'accuracy_vs_ground_truth',
    secondary: ['confidence_calibration', 'false_positive_rate'],
    guardrails: [],
  },
};

Step 2: Implement Traffic Splitting

```javascript class PromptABTest { constructor(experimentId, promptA, promptB, splitRatio = 0.5) { this.experimentId = experimentId; this.promptA = promptA; this.promptB = promptB; this.splitRatio = splitRatio; } // Deterministic assignment based on user ID assignVariant(userId) { const hash = this.hashString(userId + this.experimentId); return hash < this.splitRatio ? 'A' : 'B'; }

hashString(str) { let hash = 0; for (let i = 0; i < str.length; i++) { hash = ((hash << 5) - hash) + str.charCodeAt(i); hash = hash & hash; } return Math.abs(hash) / 2147483647; } getPrompt(userId) { return this.assignVariant(userId) === 'A' ? this.promptA : this.promptB; } }

// In your API handler: const experiment = new PromptABTest('prompt_v2_test', PROMPT_V1, PROMPT_V2, 0.5); const prompt = experiment.getPrompt(userId); const variant = experiment.assignVariant(userId); const response = await callModel(prompt, userInput);

await logExperimentResult({ experimentId: 'prompt_v2_test', variant, userId, input: userInput, output: response, timestamp: new Date().toISOString(), });

### Step 3: Statistical Analysis

---

Sample Size Planning

---

Common A/B Testing Mistakes

Mistake 1: Peeking. Checking results before reaching minimum sample size and deciding early. The early results will almost always be noise.

Mistake 2: Multiple simultaneous experiments. Running multiple prompt changes at once makes it impossible to attribute results to any specific change. Test one variable at a time.

Mistake 3: Ignoring novelty effects. Users sometimes behave differently when first encountering a new interface. Run experiments long enough (at least a week) to get past novelty.

Mistake 4: Not controlling for input distribution. If your traffic mix changes during the experiment (e.g., more mobile users, different use cases), the experiment results are confounded.