🧩 Puzzle Piece: Every transformer block contains
residual connections — the "Add" in "Add & Norm." Instead
of replacing the input, the block computes a
correction and adds it: output = input + block(input).
This lesson teaches the linear algebra behind why this works: projections,
orthogonal decomposition, and why preserving the original signal while
adding a learned correction prevents deep networks from collapsing.
A vector can be split into two parts: the part that lies along another
vector (its projection), and the part that's perpendicular (the residual).
This decomposition — splitting "what aligns" from "what's left over" — is
the mathematical idea behind one of the most important architectural
tricks in deep learning.
Projection: The Shadow of One Vector on Another
The projection of vector v onto vector
u is the component of v that points in the direction of
u:
proju(v) = (v·u / u·u) · u
Geometrically: drop a perpendicular from the tip of v to the line through
u. The projection is the vector from the origin to that foot. It's the
closest approximation of v that lies entirely along u.
Orthogonal Decomposition
Any vector v can be split into two perpendicular (orthogonal) pieces:
v = proju(v) + (v − proju(v))
The first term is along u (the projection). The second
term is perpendicular to u (the residual). These two
components are orthogonal — their dot product is zero — meaning they carry
completely independent information.
Two vectors are orthogonal when their dot product is zero.
Orthogonal vectors have no overlap, no redundancy. An orthogonal basis —
where every basis vector is perpendicular to every other — is the most
efficient coordinate system possible because each dimension captures
completely independent information.
Residual Connections: The "Add" in "Add & Norm"
In a transformer block, the output is:
output = LayerNorm(x + Sublayer(x))
where Sublayer is either attention or the feed-forward network. Instead of
output = Sublayer(x) (replacing x entirely), the block
computes x + Sublayer(x) — it adds a
correction to the original.
Why does this matter?
Gradient flow: During training, the gradient can flow
directly through the "+" path, bypassing the sublayer. This prevents the
"vanishing gradient" problem in deep networks.
Learning residuals: The sublayer only needs to learn
the difference between what the input already encodes and what
it should encode. If the identity is already good, the sublayer can
output near-zero.
Information preservation: The original signal x is
never lost — it's always there, with the sublayer's output added as a
refinement.
Residual connections decompose learning into "keep what works, fix what
doesn't."
Each layer doesn't need to rediscover everything from scratch. It only
needs to compute the correction — the residual — and add it. This is why
transformers can be stacked 96+ layers deep without the earlier layers'
information being washed away.
Layer Normalization
The "Norm" in "Add & Norm" standardizes each vector to have mean 0 and
variance 1 (with learned scale and shift parameters). This keeps vector
magnitudes from exploding or vanishing as they pass through many layers —
a practical necessity when composing dozens of transformations.
Check Your Understanding
Q1: In a residual connection output = x + f(x), what happens if the
sublayer f(x) learns to output near-zero?
A. The block fails — zero output means no learning happenedB. The block passes x through unchanged — which is a perfectly valid
thing to doC. The model crashes because division by zero occurs in layer
normD. x is multiplied by a learned scalar instead of being added
Q2: Two vectors are orthogonal. What does this guarantee about their
dot product?
A. It equals 1B. It equals 0C. It's positive but less than 1D. It's undefined
Q3: Why can transformers be stacked to 96+ layers without losing
early-layer information?
A. Each layer stores a copy of all previous layers' outputsB. Residual connections provide a direct path for information and
gradients to flow through the networkC. Attention mechanisms have infinite memoryD. Layer normalization compresses the network into fewer effective
layers
📄 Key Paper
He et al., "Deep Residual Learning for Image Recognition" (2015)
— the paper that introduced residual connections to deep learning. The
transformer inherited this idea directly.
Residual connections feel like a small implementation trick, but they're
mathematically profound. If you want to understand why deep networks
without residuals fail, or why the "+" sign is doing more work than it
looks like, ask me.