What a large language model actually predicts

Most confusion about language models comes from imagining a step that is not there. People picture the model consulting something — an index, a database, a memory of documents — and then writing an answer. That second half is real. The first half is not.

What actually happens is narrower and stranger: given the text so far, the model estimates how likely every possible next token is, picks one, appends it, and repeats. That loop is the whole of it.

One token at a time

The model reads your input as a sequence of tokens and produces a probability distribution over its entire vocabulary for what comes next. A token is selected, added to the sequence, and the model runs again on the longer sequence.

This is why the beginning of an answer constrains the rest of it. Once the model has committed to an opening clause, every later token is conditioned on that clause. A reply that starts down the wrong path rarely recovers mid-sentence, because there is no mechanism for going back.

Why length costs more than you expect

Each new token requires another pass over the whole sequence. Generation is therefore sequential in a way that reading the prompt is not, and a long answer costs meaningfully more time than a long question.

There is no lookup, so there is no "not found"

A database that lacks a row returns nothing, and your code handles the empty case. A model has no equivalent. What it learned during training is spread across its weights as statistical structure, not stored as retrievable records.

So when you ask about something it never saw, nothing reports a miss. The model does what it always does: produces the most plausible continuation. The result is an answer shaped exactly like a correct one.

Fluency and accuracy come from the same place

It is tempting to read hesitation as uncertainty and confidence as knowledge. With a language model, that instinct is actively misleading. Both the hedging and the certainty are stylistic patterns it learned, and neither is connected to whether the underlying claim holds.

This is the single most useful thing to internalise. A well-written answer is evidence about the writing, not about the facts. Verification has to come from somewhere else — a retrieval step, a tool call, a test, a human.

What this leaves you in control of

If the only thing the model conditions on is the sequence so far, then the sequence is your entire interface. Prompting is not a magic phrase; it is assembling the context that makes the answer you want the most probable continuation.

That reframing makes the practical advice obvious. Put the relevant material in the input rather than hoping the weights contain it. State the format you need instead of parsing whatever arrives. Give examples, because examples change the distribution more reliably than adjectives do.