What Is an LLM?
A precise mental model for what a large language model actually computes, why that computation is useful, and where its limits come from structurally rather than incidentally.
Most introductions to large language models start with an analogy — “autocomplete on steroids” — and stop there. That analogy is not wrong, but it is not precise enough to build systems on. If you are going to put an LLM inside an architecture with retries, fallbacks, and human review, you need a model of what it does that lets you predict where it will fail, not just what it usually gets right.
The actual computation
An LLM is a function that takes a sequence of tokens and returns a probability distribution over the next token:
tokens_so_far → model → P(next_token | tokens_so_far)Generation is this function applied repeatedly: sample a token from the distribution, append it to the sequence, run the function again. Every property people care about — coherence, style, factuality, reasoning — is an emergent side effect of this single repeated operation, not a separate subsystem the model consults.
This matters architecturally for one reason: the model has no notion of a “correct” answer, only a probable one, conditioned on everything in its context window. It does not distinguish between the token that reflects the world and the token that is simply statistically likely given the prompt. Those usually coincide, closely enough to be extremely useful. They diverge under specific, somewhat predictable conditions — long tail facts, ambiguous instructions, prompts that push probability mass toward a plausible but wrong continuation.
Where the “knowledge” actually lives
During training, the model’s parameters are adjusted so that the predicted distribution gets closer to the actual next token across a very large text corpus. Over enough data, the parameters come to encode statistical regularities of language and of the world as described in that language: grammar, facts, code idioms, argument structures.
None of this is stored as retrievable records. There is no row you can query for “what does the model know about X.” What exists is a set of weights that, when combined with a specific prompt, produce a distribution that often reflects accurate information about X. This is why LLMs can state a fact correctly in one phrasing and get it wrong in a slightly different phrasing — the underlying “knowledge” is a statistical disposition, not a lookup.
This distinction is the whole reason retrieval-augmented approaches exist: if you need an answer to be grounded in a specific, current, or private piece of text, you cannot rely on the parameters to have encoded it reliably. You have to put the text in the context window yourself.
Context window: the model’s entire working memory
Everything the model can condition on for a given generation — instructions, retrieved documents, conversation history, tool outputs — has to fit in the context window. There is no persistent memory across calls unless your application explicitly re-supplies it.
This has a direct architectural consequence: the application, not the model, owns state. If your system behaves as though the model “remembers” something from a previous turn, what is actually happening is that your application re-sent that information as part of the prompt. Understanding this early prevents a common design mistake: treating the model as a place to store facts or decisions that the system needs to remain correct over time.
What this model predicts about failure modes
Given the above, a few things follow that are useful for engineering purposes:
- Hallucination is not a bug to be patched, it is the default behavior of a probability model asked to be specific about something under-specified in its training data or prompt. Mitigations (retrieval, constrained decoding, verification steps) work by narrowing the distribution toward text you can check, not by teaching the model what “true” means.
- Longer, more specific prompts reduce variance, not just improve style. They narrow the conditional distribution and make classification-of-what’s-likely reflect what you actually want more closely.
- The model is not a good place to enforce hard constraints. “Never output field X” is a probabilistic nudge, not a guarantee, because the underlying mechanism has no concept of a hard rule — only of what tends to follow a given prompt.
That last point is the seed of a broader architectural principle worth its own article: designing deterministic boundaries around a probabilistic component. If a behavior must be guaranteed, it belongs in code that wraps the model, not in the prompt asking the model to behave.
The useful summary
An LLM is a next-token distribution estimator, applied repeatedly, whose apparent knowledge and reasoning are statistical regularities baked into its parameters rather than retrievable facts or executed logic. That framing — not “a mind” and not “a search engine” — is the one that predicts, correctly, both what it’s good at and where you’ll need to build guardrails around it.