0010 — Entropy & Cross-Entropy: How AI Measures Wrongness

Probability Pillar · ~25 minutes

🧩 Puzzle Piece: Every GPT model ever trained — GPT-1 through GPT-4, Claude, Gemini, Llama — was trained by minimizing cross-entropy loss. The model's single training objective is: "make the probability you assign to the actual next token as high as possible." Cross-entropy is the mathematical function that measures how badly the model is failing at this. If you understand cross-entropy, you understand what every language model is optimizing.
How do you measure "wrongness"? If the correct next word is "cat" and your model says P("cat") = 0.01, how wrong is that? If another model says P("cat") = 0.73, how much better is it? Cross-entropy answers this question — and it's the loss function behind every generative AI system.

Information Content: How Surprising Is an Event?

The information content (or "surprisal") of an event with probability p is:

I(p) = −log(p)

Low probability → high surprise → high information content. If P("cat") = 0.01, the surprisal is −log(0.01) ≈ 4.6 (using natural log). If P("the") = 0.73, the surprisal is −log(0.73) ≈ 0.31 — much less surprising.

Why the log? It makes information additive: the information from two independent events is the sum of their individual informations. log(a·b) = log(a) + log(b). This is the defining property of information.

Entropy: Expected Surprise

The entropy H(P) of a distribution P is the expected surprisal:

H(P) = −Σ P(x) · log P(x) = E[−log P(X)]

Entropy measures uncertainty. A fair coin (H = 1 bit) has higher entropy than a biased coin that's heads 99% of the time (H ≈ 0.08 bits). Maximum entropy = uniform distribution (most uncertain). Minimum entropy = deterministic (certain, H = 0).

Cross-Entropy: How Wrong Is Model Q About Data From P?

The cross-entropy H(P, Q) measures how surprised model Q is by data drawn from true distribution P:

H(P, Q) = −Σ P(x) · log Q(x) = −EP[log Q(X)]

In language modeling:

Training a language model = minimizing cross-entropy. The model tries to make the correct token less surprising (higher probability → lower −log(p)). If P("cat") goes from 0.01 to 0.73, cross-entropy drops from 4.6 to 0.31 — the model improved. This single number drives all of the model's learning.

Perplexity: Cross-Entropy in Human Terms

Perplexity = exp(cross-entropy). It can be interpreted as "the effective number of choices the model is choosing between." Perplexity of 10 means the model is as uncertain as if it were picking uniformly from 10 equally likely options. Lower perplexity = better model. GPT-4 achieves perplexity around 3–5 on standard benchmarks — it's effectively choosing between just a few plausible tokens.

Check Your Understanding

Q1: A model assigns P(correct) = 0.001. What is the cross-entropy contribution for this single example (using natural log)?

A. 0.001 B. −ln(0.001) ≈ 6.9 C. 1 − 0.001 = 0.999 D. 0

Q2: Model A achieves perplexity 8, Model B achieves perplexity 4 on the same test set. What does this tell you?

A. Model B is about twice as fast B. Model B is less uncertain — it's effectively choosing between 4 equally likely tokens vs. Model A's 8 C. Model A has more parameters D. Perplexity doesn't measure model quality

Q3: Why does cross-entropy use −log(p) instead of just (1−p) as the penalty?

A. −log(p) is traditional — there's no mathematical reason B. −log(p) makes information additive across independent events, explodes as p→0 (infinite penalty for impossible events), and has deep connections to coding theory and maximum likelihood C. (1−p) gives the same results but is slower to compute D. −log(p) ensures the loss is always positive
📺 Primary Source Watch: A Short Introduction to Entropy, Cross-Entropy and KL-Divergence — Aurélien Géron (11 min)
📖 Recommended Reading Blitzstein & Hwang, Introduction to Probability — skim Chapter 13 (Information Theory connections); also see Chapter 6 of Bishop's Pattern Recognition and Machine Learning
Cross-entropy is the "north star" of language model training. Every weight update during training is trying to make this one number smaller. If you ever see a training curve going down — that's cross-entropy decreasing. If you see it plateau — the model stopped learning. It's the single most important number in generative AI.
← 0009: Random Variables & Distributions Next: 0011 — KL Divergence →