0000 — The Transformer: Your North Star

Precursor · The destination before the journey · ~25 minutes

You already use transformers every day. ChatGPT, Claude, Gemini — all transformer-based. DALL·E, Midjourney, Sora — built on transformer variants. GitHub Copilot, the assistant writing code with you right now — a transformer. This lesson shows you what's inside the black box. Not the code, not the engineering — the mathematical ideas that make it work. Everything you learn in the next 14 lessons builds toward understanding this architecture from the ground up.

What Does a Transformer Do?

At its core, a transformer takes a sequence of tokens and predicts what comes next.

"The cat sat"

→

Transformer

→

"on" (73%)
"under" (11%)
"." (3%)
…

That's it. Input tokens in, probability distribution over next tokens out. Repeat to generate text: take the predicted token, append it to the input, run the transformer again.

This is the fundamental operation of every large language model. GPT-4, Claude 4, Gemini 2.5 — all of them, at their mathematical core, do exactly this: take a sequence of tokens, output a probability distribution over the next token. The difference between them is scale (billions vs. trillions of parameters), training data, and engineering — not the underlying math.

The Four Mathematical Pillars

The transformer rests on four mathematical ideas. You'll master the first two in depth, and we'll touch the third and fourth as needed:

🧮 Linear Algebra

The language of data representation and transformation

Vectors — words, images, users become lists of numbers
Matrices — transformations that move vectors between spaces
Matrix multiplication — the computational engine of attention
Embeddings — learned vector representations that capture meaning
SVD / low-rank — how we compress and understand models

Lessons 0001–0006

🎲 Probability & Information Theory

The language of uncertainty, learning, and measurement

Probability distributions — the model's output format
Conditional probability & Bayes — learning from evidence
Entropy & cross-entropy — how we measure "wrongness"
KL divergence — comparing distributions
Softmax — the bridge between vector math and probability

Lessons 0007–0012

↓

Attention Mechanism
Where linear algebra meets probability

Lessons 0013–0014

The Transformer Pipeline — A Preview

Here is the full mathematical pipeline. Don't worry about understanding it yet. Read it once to see the shape. Return to this section after every few lessons — each time, more of it will make sense.

Tokens
"The cat sat"

→

Embedding
Tokens → Vectors

→

+ Position
encoding

→

Attention
Block
× N layers

→

Output
Projection

→

Softmax
→ Probabilities

→

"on" (73%)

Zooming into one Attention Block (the heart of the transformer):

Input
vectors

→

Multi-Head
Attention

→

Add &
Norm

→

Feed-Forward
Network

→

Add &
Norm

→

Output
vectors

And zooming into Multi-Head Attention (the core innovation of the 2017 paper):

Attention(Q, K, V) = softmax (

Q·Kᵀ

/ √dₖ ) ·

Q·Kᵀ = matrix multiplication (Lesson 0003) | softmax = probability bridge (Lesson 0012) | √dₖ = scaling factor (Lesson 0013)

That formula is the destination. After 12 lessons, you'll be able to read Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V and know exactly what each symbol means, why it's there, and what would break if you changed it. That's genuine mathematical competence — the ability to reason about the formula, not just recognize it.

Your Journey: The Lesson Map

#	Lesson	Key Question Answered
0001	Vectors — The Atom of AI Data	What is a vector, and why is everything in AI a vector?
0002	Matrices as Transformations	How do we move vectors between spaces?
0003	Matrix Multiplication — The Engine	How does Q·Kᵀ compute all pairwise similarities at once?
0004	Vector Spaces, Bases & Embeddings	What does it mean to "represent" a word as a vector?
0005	Projections, Orthogonality & Residuals	Why do residual connections help deep networks?
0006	Eigenvalues, SVD & Matrix Secrets	How do we compress models and understand what matrices "do"?
0007	Probability Fundamentals	Why is probability the right language for learning from data?
0008	Conditional Probability & Bayes	How does a model update its beliefs with new evidence?
0009	Random Variables & Distributions	What exactly is the model's "output"?
0010	Entropy & Cross-Entropy	How do we measure whether a model is "wrong"?
0011	KL Divergence	How different are two probability distributions?
0012	Softmax — The Bridge	How do we turn vector math into probabilities?
0013	Attention — The Core Innovation	How does attention actually work, step by step?
0014	Complete Transformer + Paper	How do all the pieces fit together into a working system?

What You'll Be Able to Do

After completing this sequence, you will be able to:

Read the "Attention Is All You Need" paper with genuine comprehension — every equation, every diagram
Evaluate AI claims critically: when someone says "our model uses a novel attention mechanism," you'll know what questions to ask
Reason about where transformers will fail: token limits, attention bottlenecks, distribution mismatches
Communicate precisely about AI capabilities: "the model outputs a categorical distribution over 50,257 tokens" instead of "it guesses the next word"
Bridge between technical teams and decision-makers, with the math to back up every statement

You will not be able to build a transformer. That's not the goal. You'll be able to do something rarer and arguably more valuable: think clearly about what transformers can and cannot do, and explain why, in mathematical terms that are actually correct. In an AI-driven economy, that skill — genuine mathematical literacy without the implementation — is the gap this course fills.

Check Your Understanding

Q1: At its mathematical core, what does a transformer (like GPT-4) output?

A. A single best-guess word B. A probability distribution over possible next tokens C. A fixed-length vector representing the entire input D. A confidence score between 0 and 1

Q2: The attention formula includes Q·Kᵀ (matrix multiplication), a softmax, and multiplication by V. Which two mathematical pillars converge in this formula?

A. Calculus and statistics B. Linear algebra and probability C. Graph theory and optimization D. Number theory and information theory

Q3: A company claims their new AI model "doesn't use transformers." Based on what you now know, what would be the most productive question to ask?

A. "What programming language is it written in?" B. "What mathematical operations does it use to relate different parts of the input to each other?" C. "How many GPUs did you train it on?" D. "Is it better than GPT-4?"

📖 Primary Source for the Full Journey Vaswani et al., "Attention Is All You Need" (2017) — arxiv.org/abs/1706.03762 The 9-page paper that started the transformer revolution. Bookmark it. In Lesson 0014, you'll read every section with full mathematical comprehension. For now, just know it exists — it's your destination.

📺 Recommended First Watch 3Blue1Brown, "But what is a GPT? Visual intro to transformers" — YouTube An outstanding visual walkthrough of the transformer architecture. Watch it now for motivation — then watch it again after Lesson 0014 to see how much you've learned.

This precursor is your map. Return to it whenever you feel lost in the details of a specific lesson. The question is always: "How does this piece fit into the transformer?" If you can't see the connection, ask me — I'll trace the thread for you.

Next: 0001 — Vectors: The Atom of AI Data →