Why temperature changes the answer, not the knowledge
Temperature reshapes the probability distribution before a token is sampled. Low values make the model pick its top choice more often, which reads as consistency; high values flatten the distribution and admit less likely tokens. Neither setting adds knowledge or checks facts, so temperature is a control over variance, not correctness.
Temperature is the first parameter most people meet and the most commonly misread. The mental model that causes trouble is "low temperature makes the model more careful". It does not. It makes the model more predictable, and those are different properties.
What the number does
At each step the model produces scores across its vocabulary. Temperature divides those scores before they are turned into probabilities. Dividing by a small number exaggerates the gaps, so the top candidate dominates. Dividing by a larger one flattens the field and gives unlikely tokens a real chance.
Nothing in that operation consults new information. The ranking was already decided; temperature only adjusts how sharply the model commits to it.
Where the variance actually shows up
Most tokens in a fluent sentence have one overwhelming favourite, so temperature changes nothing there. The effect concentrates at the branch points — the start of a paragraph, the choice of an example, the framing of an opening clause. Which is precisely why a small change can alter a whole answer.
Repeatable is not the same as right
Set temperature to zero and run the same input twice, and you will get the same output. That is genuinely valuable: it makes evaluation meaningful, caching effective and bug reports reproducible.
What it does not do is improve the answer. If the top-ranked continuation was a confident fabrication, zero temperature guarantees you that fabrication every time. You have removed the noise, not the error.
When higher values earn their keep
Variety is the point in brainstorming, naming, copy variations and anywhere you intend to generate several options and choose. Sampling more broadly is how you get options that differ in substance rather than wording.
The cost is coherence over length. A high temperature applied to a long structured document compounds: each unusual choice conditions everything after it, and the output drifts.
The dial to reach for instead
When accuracy is the complaint, temperature is almost never the fix. The productive moves are upstream: put the source material in the context, constrain the output format, add an example of the shape you want, or add a verification step that checks the claim against something real.
Use temperature to decide how much variety you want. Use everything else to decide how right the answer is.
Frequently asked questions
Should I set temperature to zero for factual work?
It is a reasonable default for anything you want to be reproducible, such as classification or extraction. Just be clear about what you gained: the same input now yields the same output. If that output was wrong, it will be wrong every time.
Does temperature zero guarantee identical outputs?
In practice, usually but not always. Batching, hardware and floating-point non-determinism can still produce small differences between runs. Treat it as very low variance rather than a guarantee.
