Chapter 16: Chapter 16: Self-Consistency and Ensemble Techniques

The Problem With Single-Sample Decisions

When you make one API call and use the result, you are sampling once from a probability distribution. The output might be excellent. It might be one of the model's worst possibilities. You have no way to know.

For low-stakes tasks (draft blog posts, boilerplate code, initial brainstorming), single-sample is fine. For high-stakes tasks (medical classification, legal analysis, critical code paths, financial decisions), single-sample is inadequate.

Self-consistency and ensemble techniques make high-stakes AI decisions more reliable.

---

Technique 1: Self-Consistency Sampling

Generate the same prompt N times (at non-zero temperature) and aggregate the results.

This works because: if the model produces the same answer across 9 out of 10 samples, that answer is more likely to be correct than one produced in a single sample. Variance across samples reveals uncertainty. Consistency across samples indicates confidence.

javascript

async function selfConsistencyAnswer(prompt, n = 10) {
  const samples = await Promise.all(
    Array.from({ length: n }, () => 
      client.chat.completions.create({
        model: 'gpt-4o',
        messages: [{ role: 'user', content: prompt }],
        temperature: 0.7,
        response_format: { type: 'json_object' },
      })
    )
  );
  
  const answers = samples.map(s => JSON.parse(s.choices[0].message.content));
  return aggregate(answers);
}

// For classification tasks: majority vote
function aggregateClassifications(answers) {
  const counts = {};
  for (const answer of answers) {
    counts[answer.classification] = (counts[answer.classification] || 0) + 1;
  }
  const winner = Object.entries(counts).sort((a, b) => b[1] - a[1])[0];
  return {
    classification: winner[0],
    confidence: winner[1] / answers.length,
    distribution: counts,
  };
}

// For numeric estimates: median (more robust than mean)
function aggregateNumerics(answers, field) {
  const values = answers.map(a => a[field]).sort((a, b) => a - b);
  const mid = Math.floor(values.length / 2);
  return values.length % 2 ? values[mid] : (values[mid - 1] + values[mid]) / 2;
}

When to use self-consistency:

Classification tasks where you need confidence, not just a label
Numeric estimation where outlier resistance matters
Any task where a wrong answer has significant consequences
When you need a measure of the model's uncertainty

Cost: N API calls instead of 1. For high-stakes decisions, this is almost always worth it.

---

Technique 2: Multi-Model Ensemble

Use multiple different models and aggregate their outputs. This is more reliable than multiple samples from one model because different models have different error modes — they are unlikely to be wrong in the same way simultaneously.

javascript

async function ensembleAnswer(prompt) {
  const models = [
    { name: 'gpt-4o', client: openaiClient },
    { name: 'claude-opus-4-5', client: anthropicClient },
    { name: 'gemini-2.5-pro', client: geminiClient },
  ];
  
  const responses = await Promise.all(
    models.map(m => callModel(m.client, m.name, prompt))
  );
  
  if (allAgree(responses)) {
    return { answer: responses[0], confidence: 'high', consensus: true };
  } else {
    return {
      answer: await synthesiseDisagreements(responses),
      confidence: 'medium',
      consensus: false,
      individual_answers: responses,
    };
  }
}

---

Technique 3: Chain-of-Thought + Self-Consistency

The most powerful combination for complex reasoning:

Use chain-of-thought prompting to get explicit reasoning in each sample
Generate N samples with CoT
Aggregate final answers by majority vote
Use the reasoning chains to understand why answers differ

javascript

async function cotSelfConsistency(problem, n = 7) {
  const samples = await Promise.all(
    Array.from({ length: n }, () =>
      client.chat.completions.create({
        messages: [
          { role: 'system', content: 'Solve problems step by step. Show your full reasoning before giving a final answer.' },
          { role: 'user', content: problem + '\n\nThink through this step by step, then give your final answer in the format: ANSWER: [your answer]' }
        ],
        temperature: 0.8,
      })
    )
  );
  
  const parsedSamples = samples.map(s => {
    const text = s.choices[0].message.content;
    const answerMatch = text.match(/ANSWER:\s*(.+)/i);
    return { reasoning: text, answer: answerMatch ? answerMatch[1].trim() : null };
  });
  
  const answerCounts = {};
  parsedSamples.forEach(s => {
    if (s.answer) answerCounts[s.answer] = (answerCounts[s.answer] || 0) + 1;
  });
  const winner = Object.entries(answerCounts).sort((a, b) => b[1] - a[1])[0];
  
  return {
    answer: winner[0],
    confidence: winner[1] / n,
    all_answers: answerCounts,
  };
}

---

When Ensemble Is Worth the Cost

Task	Recommended Approach	Why
Casual text generation	Single sample	Low stakes, creativity valued
Document classification	Self-consistency (5 samples)	Need confidence, moderate stakes
Medical triage	Multi-model ensemble	High stakes, different model strengths
Financial analysis	CoT + self-consistency	Complex reasoning, high stakes
Code for critical paths	Multi-model + human review	Highest stakes
Creative brainstorming	Single sample (high temp)	Diversity preferred over accuracy

The rule: Stakes x Complexity = Sampling investment required.

Chapter 16: Self-Consistency and Ensemble Techniques

The Problem With Single-Sample Decisions

Technique 1: Self-Consistency Sampling

Technique 2: Multi-Model Ensemble

Technique 3: Chain-of-Thought + Self-Consistency

When Ensemble Is Worth the Cost

Finished reading this chapter?

Download Printable Chapter Checklist

The Problem With Single-Sample Decisions

Technique 1: Self-Consistency Sampling

Get advanced blueprint resources

Technique 2: Multi-Model Ensemble

Technique 3: Chain-of-Thought + Self-Consistency

When Ensemble Is Worth the Cost

Finished reading this chapter?

Download Printable Chapter Checklist