Every symbol in this formula has a precise mathematical meaning. You now know all of them:
| Symbol | What It Is | Lesson |
|---|---|---|
| Q | Query matrix — "what am I looking for?" Each row is one token's query vector | 0002 |
| K | Key matrix — "what do I contain?" Each row is one token's key vector | 0002 |
| V | Value matrix — "what information do I carry?" Each row is one token's value vector | 0002 |
| Q·Kᵀ | Matrix multiplication — all pairwise dot products between queries and keys | 0003 |
| √dₖ | Scaling factor — prevents softmax saturation in high dimensions | 0012 |
| softmax(·) | Row-wise softmax — converts similarity scores to probability distributions | 0012 |
| ·V | Weighted average of value vectors, using attention probabilities as weights | 0001, 0003 |
The input is a matrix X (seq_len × d_model) — one row per token. Three learned weight matrices project X into query, key, and value spaces:
Q = X·WQ K = X·WK V = X·WV
Each token now has three representations: what it's looking for (query), what it offers (key), and what it carries (value).
Scores = Q·Kᵀ
Entry (i,j) = qᵢ·kⱼ — how relevant token j's content is to token i's query. This is an (seq_len × seq_len) matrix of raw similarity scores.
Weights = softmax(Scores / √dₖ)
Each row becomes a probability distribution over all tokens. Row i tells you: "token i should pay X% attention to token 0, Y% to token 1, Z% to token 2…" The weights in each row sum to 1.
Output = Weights · V
Each output row is a weighted average of all value vectors, where the weights are the attention probabilities. Token i's output = Σⱼ (attention weight ij) · vⱼ.
Tokens that got high attention contribute more to the output. Tokens that got near-zero attention contribute almost nothing. The output at position i is a context-aware representation of token i — it's been enriched with information from every relevant token in the sequence.
Instead of one set of Q/K/V projections, the transformer uses multiple heads (typically 8–128):
MultiHead(Q,K,V) = Concat(head₁, ..., headₕ) · WO
where headᵢ = Attention(Q·WQⁱ, K·WKⁱ,
V·WVⁱ)
Each head learns to attend to different things: one head might track syntactic relationships (subject-verb), another might track semantic relationships (synonyms), another might track positional patterns. The heads operate in parallel, and their outputs are concatenated and projected back to the model dimension.
During autoregressive generation, the model shouldn't see future tokens. A causal mask sets all scores where j > i to −∞ before softmax:
MaskedScoresij = { Scoresij if j ≤ i; −∞ if j > i }
After softmax: exp(−∞) = 0, so future tokens get exactly zero attention weight. The model can only attend to the current token and previous tokens.
Q1: In attention, why are there three separate projections (Q, K, V) instead of just one?
A. Three is an arbitrary choice from the original paper B. Q and K are for computing relevance (via dot product), while V is what actually gets aggregated. Separating them lets the model learn different representations for "matching" vs. "content to transmit" C. It makes the model three times larger D. Q and K are redundant — one projection would work the sameQ2: After softmax, the attention weights for token i sum to 1 across all tokens j. What does this guarantee?
A. Token i attends equally to all tokens B. The output for token i is a convex combination (weighted average) of value vectors — it stays in the same "ballpark" as its inputs C. The model can only attend to one token at a time D. The output is always the same as the inputQ3: A startup claims their new architecture "replaces attention with a more efficient mechanism." What mathematical question should you ask?
A. "What's your GitHub star count?" B. "How does your mechanism compute relationships between tokens — does it use pairwise dot products (like attention), convolutions, recurrences, or something else? And what's the computational complexity in sequence length?" C. "Is it faster than a GPU?" D. "Does it use Python?"