0014 β€” The Complete Transformer + Reading the Paper

Convergence Β· ~35 minutes Β· Final lesson

🧩 Puzzle Piece: This is the summit. Every concept from the previous 13 lessons assembles into the complete transformer architecture. By the end of this lesson, you will be able to read "Attention Is All You Need" β€” the 9-page paper that launched the modern AI era β€” with genuine mathematical comprehension. Every equation, every diagram, every architectural decision will be in your vocabulary.
You started 14 lessons ago with "what is a vector?" Now you're about to understand, in full mathematical detail, the architecture that powers ChatGPT, Claude, Gemini, DALLΒ·E, Sora, and GitHub Copilot. Let's assemble the pyramid.

The Complete Architecture

Input: "The cat sat" β†’ Token IDs: [464, 3797, 6313]

β–Έ Embedding Layer: token IDs β†’ vectors (d_model = 768) [Lesson 0004]
β–Έ + Positional Encoding: add position signal [Lesson 0004]

↓ Repeat N=12 times ↓

β–Έ Multi-Head Attention: tokens exchange information [Lesson 0013]
β–Έ Add & Layer Norm: residual + stabilize [Lesson 0005]
β–Έ Feed-Forward Network: per-token transformation
β–Έ Add & Layer Norm: residual + stabilize [Lesson 0005]

↓ After final layer ↓

β–Έ Output Projection: final hidden state β†’ logits (vocab_size = 50,257)
β–Έ Softmax: logits β†’ probability distribution [Lesson 0012]

Output: P("on")=0.73, P("under")=0.11, P(".")=0.03, …

Component-by-Component Walkthrough

1. Tokenization + Embedding (Input β†’ Vectors)

Text is split into tokens (subwords). Each token ID is looked up in the embedding matrix E (vocab_size Γ— d_model). The result is a matrix X (seq_len Γ— d_model). Positional encodings are added (not concatenated) β€” this is why the dimensions must match. Lessons 0001, 0004.

2. Multi-Head Attention (The Core)

Each token's vector is projected into Q, K, V through learned matrices. Attention computes context-aware representations: each token gathers information from all other tokens, weighted by relevance. Multiple heads run in parallel, capturing different relationship types. Lesson 0013.

3. Add & Norm (Residual + Stabilization)

Residual connection: output = LayerNorm(x + Attention(x)). The "+" preserves the original signal; LayerNorm prevents values from growing/shrinking across layers. Without residuals, deep transformers don't train. Lesson 0005.

4. Feed-Forward Network (Per-Token Processing)

A small two-layer neural network applied independently to each position:

FFN(x) = ReLU(xΒ·W₁ + b₁)Β·Wβ‚‚ + bβ‚‚

Typically, the hidden dimension is 4Γ— d_model (e.g., 768 β†’ 3072 β†’ 768). This is where most of the model's parameters live β€” the FFN layers contain ~β…” of all parameters. Lessons 0002, 0003.

5. Stack N Layers

The block (Attention β†’ Add&Norm β†’ FFN β†’ Add&Norm) is repeated N times (N=12 for GPT-2 base, N=96 for GPT-3). Each layer refines the representations. Deeper layers capture more abstract patterns. Lesson 0005.

6. Output: Hidden State β†’ Probabilities

The final hidden state at the last position is multiplied by the unembedding matrix (often tied to the embedding matrix: W_unembed = Eα΅€). This produces logits β€” a vector of length vocab_size. Softmax converts logits to a probability distribution. The model samples from this distribution to generate the next token. Lessons 0007–0012.

Training: How It Learns

During training (not inference), the full forward pass happens, then:

  1. Loss computation: Cross-entropy between predicted distribution Q and true next token P: loss = βˆ’log Q(correct_token). Lessons 0010, 0011.
  2. Backpropagation: Gradients flow backward through every operation β€” softmax, matrix multiplications, embeddings, all of it. The chain rule from calculus (your calculus background!) computes how each parameter contributed to the loss.
  3. Gradient descent: Every parameter (all ~175B of them) is nudged slightly in the direction that reduces loss. The learning rate controls how big the nudge is.
  4. Repeat billions of times across the entire training corpus.
Training is just: forward pass β†’ compute loss (cross-entropy) β†’ backward pass (chain rule) β†’ update parameters (gradient descent) β†’ repeat. All the mathematical machinery you've learned β€” vectors, matrices, softmax, cross-entropy β€” is designed to make this loop work efficiently and at scale. The transformer architecture isn't just clever; it's optimized for gradient-based learning.

Reading "Attention Is All You Need"

You are now ready to read the original transformer paper with genuine comprehension. Here's a section-by-section guide:

Section What to Look For Your Math
Β§1 Introduction Motivation: replacing recurrence with attention β€”
Β§2 Background Prior work, why attention is different β€”
Β§3.1 Scaled Dot-Product Attention The core formula. Recognize Q, K, V, softmax, √dβ‚– 0001–0003, 0012–0013
Β§3.2 Multi-Head Attention Parallel heads, concatenation, W_O projection 0002–0004, 0013
Β§3.3 Position-wise FFN Two linear transformations with ReLU 0002–0003
Β§3.4 Embeddings + Softmax Embedding matrix, weight tying, output softmax 0004, 0012
Β§3.5 Positional Encoding Sine/cosine functions, why addition not concatenation 0001, 0004
Β§4 Why Self-Attention Complexity analysis: O(nΒ²Β·d) per layer 0003
Β§5 Training Optimizer (Adam), regularization (dropout, label smoothing) 0007–0011
Β§6 Results BLEU scores, state-of-the-art at the time β€”
The entire paper is 9 pages. Section 3 (Model Architecture) is 3 pages. You now have the mathematical vocabulary to read every equation in those 3 pages. The symbols Q, K, V, softmax, √dβ‚–, the matrix multiplication notation, the attention formula β€” they're all in your toolkit. You've earned this.

Beyond Transformers: What Comes Next

The transformer is the foundation, not the end. Here's how your mathematical literacy extends forward:

You can now learn any AI architecture. The mathematical foundations you've built β€” linear algebra for representation and transformation, probability for uncertainty and learning β€” are the universal language of AI. Every new architecture (Mamba, Jamba, RWKV, Hyena, Monarch Mixer) is described in this language. You've learned to read, not just to recognize.

Your 14-Lesson Journey

Check Your Understanding

Q1: The feed-forward network in a transformer block is applied independently to each position. Why is this okay, given that attention already mixed information across positions?

A. It's a bug that was never fixed B. Attention handles cross-position mixing; the FFN handles per-position transformation. The division of labor is clean: attention = "communicate", FFN = "think" C. The FFN actually mixes positions internally D. Position-independent processing doesn't work β€” this is why transformers need many layers

Q2: The embedding matrix and the output projection matrix are often "tied" (W_output = Eα΅€). Why does this make mathematical sense?

A. It saves memory β€” no deeper reason B. The embedding maps tokens β†’ vectors; the output maps vectors β†’ token probabilities. If a token's embedding is a good representation of its meaning, then using the same mapping in reverse to score how well a vector matches each token is mathematically natural C. The matrices are forced to be tied by the transformer architecture D. Tying matrices always improves performance

Q3 (Capstone): A new paper claims to improve transformers by replacing softmax in attention with a different normalization. What do you now know to ask about this claim?

A. "What programming language did you use?" B. "Does your alternative still produce row-stochastic matrices (non-negative, rows sum to 1)? Does it preserve the order of attention scores? Is it differentiable? Does it avoid saturation like √dβ‚– scaling does? And does it maintain or improve the convex combination property that stabilizes training?" C. "Is it faster than softmax?" D. "Does it use fewer lines of code?"
πŸ“„ Read Now: The Paper Vaswani et al., "Attention Is All You Need" (2017) β€” arxiv.org/abs/1706.03762
Start with Section 3. Read it aloud if it helps. You know every symbol. You understand every operation. This paper is now written in your language.
πŸ“Ί Recommended Victory Lap Re-watch: 3Blue1Brown, "But what is a GPT? Visual intro to transformers" β€” YouTube
Watch it again now. The first time (Lesson 0000), you saw shapes and ideas. This time, you'll see QΒ·Kα΅€, softmax, √dβ‚–, residuals, embeddings β€” the actual mathematics, animated. The difference is what you've learned.
You did it. 15 lessons (0000–0014), two pillars of mathematics, one architecture. You can now read AI papers, evaluate AI claims, and reason about AI systems in the language they're actually built in β€” linear algebra and probability. The goal was never to build a transformer. It was to understand one. You've achieved that. What you do with this literacy β€” which domain you apply it to, which opportunities you recognize, which decisions you inform β€” that's your mission now.
← 0013: The Attention Mechanism Return to 0000: The North Star (re-read with new eyes)