0002: Matrices as Transformations

🧩 Puzzle Piece: In the transformer, the attention mechanism applies three matrix transformations to every token vector: Q (query), K (key), and V (value) projections. A matrix is a machine that takes a vector and produces a new vector — transforming representation from one space to another. This lesson teaches you what that means.

You've seen vectors — arrows in space. Now: what if you had a machine that could take any arrow, rotate it, stretch it, shear it, or flip it — all with a single operation? That machine is a matrix.

What Is a Matrix?

A matrix is a rectangular grid of numbers. A 2×3 matrix has 2 rows and 3 columns:

But a matrix isn't just a table of numbers. A matrix is a function — it takes vectors as input and produces vectors as output. Specifically, multiplying a matrix by a vector applies a linear transformation to that vector.

Matrix × Vector: The Operation

To multiply a matrix A (m×n) by a vector x (n×1), each row of A takes a dot product with x:

┌ ┐ ┌ ┐   ┌ ┐
│ a b c │ │ x │  │ ax + by + cz │
│ d e f │ │ y │ = │ dx + ey + fz │
└ ┘ │ z │  │ gx + hy + iz │
          └ ┘  └ ┘

Each output component is a weighted combination of all input components. The matrix tells you how to mix the inputs to get the outputs.

Key insight: Matrix × vector = linear combination. Each row of the matrix defines a "recipe" — how much of each input component to use. This is very close to what a neural network layer does: take inputs, apply weighted combinations, produce outputs. The "weights" in a neural network are literally the entries of a matrix.

What Transformations Look Like

In 2D, we can visualize what matrices do to the entire plane. Here are the fundamental transformations:

Matrix	Transformation	What It Does
`┌2 0┐ │0 2│ └ ┘`	Uniform scaling	Stretches everything by ×2
`┌2 0┐ │0 1│ └ ┘`	Axis scaling	Stretches x-axis only — squares become rectangles
`┌0 −1┐ │1 0│ └ ┘`	90° rotation	Rotates every vector counterclockwise by 90°
`┌−1 0┐ │0 −1│ └ ┘`	Reflection	Flips everything through the origin
`┌1 1┐ │0 1│ └ ┘`	Shear	Tilts the plane — vertical lines slant
`┌1 0┐ │0 0│ └ ┘`	Projection	Collapses everything onto the x-axis

Every linear transformation is a matrix. Every matrix is a linear transformation. This equivalence — that a grid of numbers and a geometric operation are the same thing — is one of the most powerful ideas in applied mathematics. It means you can think geometrically (rotate, stretch, shear) but compute algebraically (multiply rows by columns).

Why This Matters for the Transformer

In the attention mechanism, every token vector x goes through three matrix transformations:

Each W is a learned matrix — its numbers are discovered during training. W_Q transforms x into a "query" (what am I looking for?), W_K into a "key" (what do I contain?), and W_V into a "value" (what information do I carry?).

Same vector, three different transformations, three different purposes. The matrices are the model's knowledge — they encode what the model has learned about how to represent and relate information.

When people say a model has "175 billion parameters," they're counting matrix entries. GPT-3's 175B parameters are literally the numbers inside its matrices. The matrices are the model. Understanding matrices as transformations means understanding what those 175 billion numbers actually do.

Matrix Multiplication Preview

What happens when you apply one transformation, then another? You get a composition of transformations — and that composition is itself a matrix. Matrix × matrix = composition.

If B rotates and A scales, then A·B first rotates then scales — in one matrix. This is why matrix multiplication matters so much: it's how we combine transformations. That's the next lesson.

Check Your Understanding

Q1: In the attention mechanism, the matrix W_Q transforms a token vector x into a query vector q. What is W_Q, fundamentally?

A. A lookup table that maps tokens to queries B. A learned linear transformation — a grid of numbers discovered during training C. A fixed mathematical constant like π D. A random number generator

Q2: A 2×2 matrix with zeros everywhere except a 3 in the top-left position transforms the vector [1, 1]. What is the result?

A. [1, 1] B. [3, 0] C. [3, 3] D. [0, 0]

Q3: If a model has "7 billion parameters," what are those parameters?

A. Hyperparameters like learning rate and batch size B. The entries of the model's matrices — the numbers that define its transformations C. The number of tokens in the training data D. The number of different tasks the model can perform

📺 Primary Source Watch: Linear transformations and matrices — 3Blue1Brown (Essence of Linear Algebra, Ch. 3) The best 10-minute geometric intuition for matrices as transformations you will ever see.