Two probability distributions over the same set of outcomes β how
different are they? Not just "are they the same peak?" but "how much
information would I lose if I used one instead of the other?"
KL divergence answers this β and it's the theoretical
foundation of AI training.
KL divergence measures the expected log ratio of P to Q,
where the expectation is taken under P (the true distribution).
Intuitively: for each outcome x, compute how much more probable P thinks x
is than Q thinks x is, in log space, then average over P.
Key Properties
KL(P β Q) β₯ 0 β always non-negative. Zero if and only
if P = Q everywhere.
KL(P β Q) β KL(Q β P) β it's asymmetric! This is
critical. Using Q to approximate P is not the same as using P to
approximate Q.
Not a true distance metric β because of the asymmetry
and because it doesn't satisfy the triangle inequality. It's a
"divergence."
The asymmetry matters for AI. KL(P_data β Q_model)
penalizes the model for assigning low probability where the data has high
probability (mode-covering). KL(Q_model β P_data) penalizes the model for
assigning high probability where the data has low probability
(mode-seeking). Different training objectives optimize different KL
directions, leading to different model behaviors (e.g., beam search vs.
sampling).
Cross-Entropy = Entropy + KL Divergence
H(P, Q) = H(P) + KL(P β Q)
Cross-entropy decomposes into: the
unavoidable uncertainty of the data itself (H(P)) plus
the extra uncertainty from using the wrong model (KL
divergence).
During training, H(P) β the entropy of real language β is constant (it's a
property of the data, not the model). So
minimizing cross-entropy = minimizing KL divergence. The
model can't make language less inherently uncertain; it can only reduce
the gap between its predictions and reality.
This completes the training story. The model's loss
function is cross-entropy = H(data) + KL(data β model). Since H(data) is
fixed, minimizing the loss means minimizing KL divergence β making the
model's distribution as close as possible to the true distribution of
language. That's what "learning" means mathematically: reducing KL
divergence.
Check Your Understanding
Q1: KL(P β Q) = 0. What does this imply?
A. P and Q are completely different distributionsB. P = Q for all outcomes β the distributions are identicalC. Q is a uniform distributionD. P has zero entropy
Q2: During training, cross-entropy loss decreases from 4.2 to 2.1.
What's happening mathematically?
A. The entropy of language H(P) is decreasingB. KL(P β Q_model) is decreasing β the model's distribution is
getting closer to the true data distributionC. The model is memorizing the training data exactlyD. The vocabulary size is shrinking
Q3: A model assigns P("the") = 0.9 when the true probability in
language is 0.3. Another model assigns P("the") = 0.3. Which
contributes more to KL divergence, and why?
A. The 0.9 model β because 0.9 > 0.3B. The 0.9 model β because KL uses the log ratio: log(0.3/0.9) β
β1.1, weighted by P=0.3C. They contribute equallyD. Neither β KL divergence only penalizes underestimation, not
overestimation
π Recommended Reading
Blitzstein & Hwang, Introduction to Probability β Section 13.2
(Entropy and KL); Bishop,
Pattern Recognition and Machine Learning β Section 1.6
(Information Theory)
This completes the probability pillar. The next lesson β Softmax β is
where both pillars fuse. You now have linear algebra (vectors, matrices,
dot products, transformations) and probability (distributions, entropy,
KL). The bridge between them is the softmax function.