"The probability that it will rain tomorrow" and "the probability that it
will rain tomorrow, given that it's cloudy today" are different questions.
The second is conditional β and it's almost always the
better question. AI models don't answer "what's the most common word?"
They answer "what's the most likely word
given everything said so far?"
In words: the probability of A given B is the fraction of
B's probability mass that also satisfies A. You're zooming in on the part
of the sample space where B is true, and measuring A's share of that
subspace.
Example: P("on" | "The cat sat") vs. P("on" | "The cat
sat on the") β the second probability should be near-zero (you don't say
"on the on the mat"), even though "on" is a common word. The context
changes everything.
Bayes' Theorem
P(A | B) = P(B | A) Β· P(A) / P(B)
Bayes' theorem tells you how to update your beliefs when
you observe evidence:
P(A): prior β what you believed before seeing evidence
P(B | A): likelihood β how probable is the evidence if
A is true?
P(A | B): posterior β what you should believe after
seeing evidence
Bayes' theorem is the mathematical engine of learning.
Start with prior beliefs β observe data β compute posterior (updated
beliefs). Repeat with new data. This iterative updating β prior β evidence
β posterior β (becomes new prior) β more evidence β β¦ β is exactly how
both humans and AI systems learn from experience.
Autoregressive Decomposition: The Chain Rule
How does a language model compute P("The cat sat on the mat")? By breaking
it into conditional probabilities:
Each factor is exactly what the transformer computes at each step: the
probability of the next token given all previous tokens. The
joint probability of the entire sequence is the product of these
conditional probabilities. This is the
chain rule of probability, and it's why autoregressive
generation works.
Generation is just repeated conditional probability. Step
1: P(tokenβ). Step 2: P(tokenβ | tokenβ). Step 3: P(tokenβ | tokenβ,
tokenβ). The transformer's entire job, at every position, is to compute
one conditional probability distribution. Then you sample from it, append
the result, and repeat.
Check Your Understanding
Q1: P(A | B) = 0.8 and P(A) = 0.3. What does this tell you?
A. A and B are independentB. B makes A much more likely β B is strong evidence for AC. A causes BD. The numbers are inconsistent β P(A|B) can't exceed P(A)
Q2: A transformer computes P("mat" | "The cat sat on the"). Which
mathematical concept does this directly use?
A. Matrix multiplicationB. Conditional probabilityC. Eigenvalue decompositionD. Gradient descent
Q3: Why does autoregressive generation use the chain rule P(wβ,...,wβ)
= β P(wα΅’ | wβ,...,wα΅’ββ) instead of trying to compute the joint
probability directly?
A. The chain rule is faster to typeB. Direct joint probability over all possible sequences of length n
would require |V|βΏ entries β impossibly large. The chain rule reduces
it to n conditional distributions over |V| tokens eachC. The chain rule gives more accurate resultsD. It's a legal requirement for AI systems
π Recommended Reading
Blitzstein & Hwang, Introduction to Probability β Chapter 2
(Conditional Probability)
Conditional probability is the "|" in P(token | context). If you
internalize nothing else from probability, internalize this: the vertical
bar means "zooming in on the world where the right side is true."
Everything in AI is conditional on something.