Chapter 8: Temperature, Top-P, and Sampling — The Parameters That Change Everything

What 'Temperature' Actually Means

When a language model generates text, it does not simply pick the most probable next token. It samples from a probability distribution over all possible tokens. Temperature is the parameter that controls the shape of that distribution.

Low temperature (0.0-0.3): The probability distribution is sharpened — high-probability tokens become even more dominant. The model almost always picks the most likely token. Output is deterministic, consistent, and conservative.

High temperature (0.8-1.5): The probability distribution is flattened — lower-probability tokens have a much greater chance of being selected. Output is diverse, creative, and surprising, but also less accurate and less reliable.

Temperature = 0: Completely deterministic (the model always picks the highest-probability token). Repeated calls with identical inputs produce identical outputs.

This is not an abstraction. It is the mechanism that underlies every output behavior difference you observe.

---

The Task-to-Temperature Decision Matrix

---

Top-P (Nucleus Sampling): The Complementary Parameter

Top-P sampling works differently from temperature. Instead of scaling the whole distribution, it draws only from the smallest set of tokens whose combined probability exceeds P.

Top-P = 0.9: The model only samples from the top tokens that collectively represent 90% of the probability mass.

Top-P = 0.1: The model only samples from the very highest-probability tokens (a very narrow nucleus).

The Interaction Between Temperature and Top-P

Use one, not both. Setting both simultaneously makes the sampling behavior harder to predict. Most practitioners recommend:

• If you care about accuracy/consistency: lower temperature, keep Top-P at default (1.0)
• If you care about creativity/diversity: raise temperature, keep Top-P at default

---

Frequency Penalty and Presence Penalty

These two parameters prevent repetition — one of the most common failure modes in long AI-generated outputs.

Frequency penalty (0.0-2.0): Reduces the probability of each token in proportion to how many times it has already appeared. Higher values = stronger penalty for using the same words repeatedly.

Presence penalty (0.0-2.0): Applies a flat penalty to any token that has appeared at all. Encourages the model to introduce new concepts and vocabulary.

Practical guidance:

• For code: use frequency_penalty = 0. Repetition in code (variable names, patterns) is often correct.
• For long-form writing: frequency_penalty = 0.3-0.5 prevents repeating the same phrases.
• For brainstorming: presence_penalty = 0.3-0.6 encourages genuinely diverse ideas.

Practical Configurations for Common Tasks

High-accuracy code generation:

{
  model: 'gpt-4o',
  temperature: 0.1,
  top_p: 1.0,
  frequency_penalty: 0.0,
  presence_penalty: 0.0,
  max_tokens: 4000
}

Blog post / long-form content:

{
  model: 'gpt-4o',
  temperature: 0.65,
  top_p: 1.0,
  frequency_penalty: 0.35,
  presence_penalty: 0.1,
  max_tokens: 2000
}

Diverse brainstorming (10 unique ideas):

{
  model: 'gpt-4o',
  temperature: 0.95,
  top_p: 0.95,
  frequency_penalty: 0.5,
  presence_penalty: 0.4,
  max_tokens: 1000
}

Data extraction / classification:

{
  model: 'gpt-4o',
  temperature: 0.0,
  top_p: 1.0,
  frequency_penalty: 0.0,
  presence_penalty: 0.0,
  response_format: { type: 'json_object' }
}

---

The Seed Parameter: Reproducibility

For debugging and reproducible benchmarks, most APIs support a seed parameter. When the same seed is used with the same inputs and settings, the model produces the same output:

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [...],
  temperature: 0.7,
  seed: 42, // Same seed = reproducible output
});

Use seeds when running automated tests on prompt quality, so you can compare changes to your prompt against a fixed baseline.

---

Key Takeaway

Temperature is not a creativity dial. It is a precision vs. diversity trade-off. For tasks where one right answer exists, use low temperature. For tasks where many good answers exist and diversity is valuable, use higher temperature. Everything else follows from that principle.