A random variable X is a function from the sample space Ī© to real numbers. For language models, X maps each possible next token to... well, the token itself (or its ID). The key idea: X has a distribution ā a description of how probable each possible value is.
The expected value E[X] is the probability-weighted average of all possible values:
E[X] = Ī£ x Ā· P(X = x) (discrete) E[X] = ā« x Ā· f(x) dx (continuous)
Think of expectation as the center of mass of the probability distribution. If you placed weights equal to probabilities at each possible value, the expectation is where the distribution would balance.
| Distribution | What It Models | Where in AI |
|---|---|---|
| Categorical | One of K categories | Next-token prediction (K = vocab size) |
| Bernoulli | Yes/no (success/failure) | Binary classification, dropout masks |
| Gaussian (Normal) | Continuous values around a mean | Weight initialization, noise, latent spaces (VAEs, diffusion) |
| Multinomial | Counts across K categories | Bag-of-words, topic models |
The categorical distribution is the most important for transformers: it's a probability vector of length K (the vocabulary size), where each entry pi ā„ 0 and Ī£pi = 1. This is exactly what softmax produces.
For continuous random variables, probability is described by a probability density function (PDF) f(x). The probability of falling in an interval [a,b] is ā«āįµ f(x) dx. This uses your calculus background ā integration as "area under the curve."
The Gaussian (normal) distribution N(Ī¼, ĻĀ²) is the most important continuous distribution in AI:
f(x) = (1/ā(2ĻĻĀ²)) Ā· exp(ā(xāĪ¼)Ā²/(2ĻĀ²))
It appears in weight initialization (Kaiming/Xavier), variational autoencoders, diffusion models, and as the limiting distribution of sums (Central Limit Theorem). The bell curve is everywhere.
Q1: A fair die has E[roll] = 3.5. What does 3.5 represent, given that you can never actually roll a 3.5?
A. The most likely outcome B. The long-run average if you rolled infinitely many times C. The median outcome D. An error in the calculationQ2: A transformer's output for the next token is best described as what?
A. A single token ID B. A categorical random variable ā a distribution over the vocabulary C. A Gaussian random variable D. A deterministic function of the inputQ3: When people say "the model was trained to minimize the loss," what mathematical object is the loss?
A. The variance of the model's outputs B. The expected negative log probability of the correct answers ā an expectation over the training data C. The number of incorrect predictions D. The determinant of the weight matrix