Chapter 13: Evaluating and Scoring Prompt Quality

The Measurement Gap

Most prompt engineering happens by feel. You run a prompt, it looks good, you ship it. Later you change the prompt, it still looks good, you ship the change. But was the change actually better? By how much? For which inputs?

Without measurement, you cannot answer these questions. And without answers, you cannot improve systematically.

This chapter covers the methods for measuring prompt quality — from simple human evaluation to automated LLM-as-judge pipelines.

---

Defining What 'Good' Means for Your Task

Before you can measure quality, you must define it for your specific task. Quality is not universal — it depends on what you are trying to produce.

For each task type, define:

• The criteria (what dimensions matter?)
• The weight (how much does each dimension matter relative to the others?)
• The threshold (what is "good enough" vs. "needs work"?)

---

Evaluation Method 1: Human Evaluation (Ground Truth)

Human evaluation is the gold standard. It is slow and expensive, but it is the only method that captures genuine quality.

Blind comparison protocol:

• Generate output with Prompt A for 20 diverse test inputs
• Generate output with Prompt B for the same 20 inputs
• Show outputs side-by-side to a human evaluator (randomly ordered, labels hidden)
• Evaluator rates: "A is better / B is better / They are equal"
• Calculate A's win rate

Single-output scoring:

Rate this output on each dimension from 1-5:
- Correctness (does it fulfill the request?): ___
- Quality (is it actually good, not just technically correct?): ___
- Format (does it follow the required format?): ___
- Safety (are there any concerning elements?): ___

Collect 5+ ratings per output and average them.

---

Evaluation Method 2: LLM-as-Judge

For high-volume evaluation, use a model (typically a stronger one than you are evaluating) to judge output quality. This is not as reliable as human judgment but is far faster and cheaper.

```javascript async function judgeOutput(task, input, output) { const judgePrompt = [

'You are an objective evaluator assessing the quality of an AI response.', '', 'TASK DESCRIPTION: ' + task, 'USER INPUT: ' + input, 'AI RESPONSE TO EVALUATE: ' + output, '', 'Evaluate on these criteria (score each 1-5):', '1. Completeness: Does it fully address the request?', '2. Accuracy: Is the information/code correct?', '3. Format: Does it follow the expected format?', '4. Quality: Is this a genuinely useful, high-quality response?', '', 'Output JSON only: { "completeness": 1-5, "accuracy": 1-5, "format": 1-5, "quality": 1-5, "overall": 1-5, "reasoning": "string" }' ].join('\n');

const response = await judgeClient.chat.completions.create({ model: 'gpt-4o', messages: [{ role: 'user', content: judgePrompt }], response_format: { type: 'json_object' }, temperature: 0.0, }); return JSON.parse(response.choices[0].message.content); }

**Known biases in LLM-as-judge:** Models tend to prefer longer responses and responses that match their own output style. Calibrate your judge by running it against human-labelled examples and checking alignment.

---

## Evaluation Method 3: Unit Testing for Prompts

For tasks with objectively correct outputs, write automated tests:

async function runPromptTests(prompt) { let passed = 0; let failed = 0; for (const test of promptTests) { const output = await callModel(prompt, test.input); const parsed = JSON.parse(output); const allPassed = test.assertions.every(assertion => { try { return assertion(parsed); } catch { return false; } }); if (allPassed) { console.log('PASS: ' + test.name); passed++; } else { console.log('FAIL: ' + test.name); failed++; } } console.log('Passed: ' + passed + '/' + (passed+failed)); } ```

---

Building a Prompt Regression Test Suite

Every time a prompt fails in production (wrong format, hallucinated content, missed instruction), add that failure case to your test suite. This prevents regression when you update the prompt.

Your test suite grows over time into a comprehensive guard against known failure modes. A prompt that passes all regression tests has a measurable quality floor.

---

The Evaluation Dashboard

Track these metrics for every production prompt over time:

• Pass rate (automated tests): Target >95%
• LLM judge average score: Track weekly
• Human preference rate (periodic A/B): Track monthly or on major changes
• Failure mode distribution: Which of the 5 failure modes appears most?