0006: Eigenvalues, SVD & Matrix Secrets

🧩 Puzzle Piece: The weight matrices inside a transformer are large (e.g., 4096×4096). Can we compress them without losing what they've learned? Can we decompose them to understand what they "really do"? The answers come from eigenvalues and the Singular Value Decomposition (SVD). This lesson covers the ideas behind techniques like LoRA (Low-Rank Adaptation) — the dominant method for efficiently fine-tuning large language models — and why "low-rank" is a concept that matters for every AI practitioner.

A matrix is a transformation of space. But what does that transformation actually do? Rotate? Stretch? Collapse? The secret: every matrix can be decomposed into rotation → scaling → rotation. This is the Singular Value Decomposition, and it's the universal skeleton key for understanding matrices.

Eigenvectors and Eigenvalues

When a matrix A multiplies a vector v, most vectors change direction. But some special vectors — eigenvectors — only get scaled:

The vector v is an eigenvector. Its direction is unchanged by A. The scalar λ (lambda) is the eigenvalue — how much v is stretched (λ > 1), shrunk (0 < λ < 1), flipped (λ < 0), or annihilated (λ = 0).

Eigenvectors reveal the intrinsic axes of a transformation — the directions that the matrix treats as special.

If a matrix has a zero eigenvalue, it's collapsing some dimension of space. Information is being lost. This matters for AI: when weight matrices have near-zero eigenvalues, they're effectively lower-rank than their dimensions suggest. This is why compression techniques work.

Singular Value Decomposition (SVD)

Not every matrix has a full set of eigenvectors. But every matrix has an SVD:

In words: every linear transformation is: rotate → scale each axis independently → rotate again. The singular values tell you how much each axis gets stretched. If σₖ is tiny, the k-th dimension barely matters — the matrix is effectively lower-rank.

The SVD is the universal decomposition. It works for every matrix — square or rectangular, invertible or not. The singular values tell you the matrix's "effective dimension." If only 32 out of 4096 singular values are large, the matrix is effectively rank-32 — and you can compress it dramatically.

Low-Rank Approximation and LoRA

Keep only the top-k singular values (and corresponding columns of U and V). You get the best possible rank-k approximation of the original matrix, measured by how much information is preserved.

LoRA (Low-Rank Adaptation) exploits this for fine-tuning: instead of updating a full weight matrix W (d×d, millions of parameters), LoRA learns two small matrices A (d×r) and B (r×d) where r ≪ d, and the update is:

Since r might be 8 or 16 while d is 4096, the number of trainable parameters drops from d² to 2dr — a reduction of 99.6%+. This is how individuals can fine-tune models like Llama on a single GPU. The mathematical bet: the changes needed during fine-tuning are low-rank — they live in a small subspace of the full weight matrix.

LoRA works because fine-tuning adjustments are low-rank. The pre-trained model already knows a lot. Adapting it to a new task only requires small, structured changes — changes that can be represented as the product of two skinny matrices. This is a direct application of the low-rank approximation principle from SVD.

Check Your Understanding

Q1: An eigenvector v of matrix A satisfies A·v = λ·v. What does λ=0 imply?

A. v is unchanged by A B. A collapses v to the zero vector — v is in the nullspace of A C. A reflects v through the origin D. v is the only eigenvector of A

Q2: LoRA fine-tunes a 4096×4096 weight matrix using two matrices A (4096×16) and B (16×4096). How many trainable parameters does this use vs. full fine-tuning?

A. Same as full fine-tuning (16.8M) B. ~131K vs. ~16.8M — about 0.8% of the full parameters C. ~8.4M — about half D. ~2M — about 12%

Q3: An AI startup claims their model is "rank-64" in its attention matrices. What does this most likely mean?

A. The model has exactly 64 attention heads B. The weight matrices have been compressed or designed so that only 64 independent directions carry significant information C. The model was trained on exactly 64 GPUs D. The model processes exactly 64 tokens at a time

This completes the linear algebra pillar. The probability pillar starts next. If eigenvalues or SVD feel abstract, remember: they're tools for answering "what does this matrix actually do?" — and in AI, the matrices ARE the model. Understanding matrix structure means understanding model structure.

0006 — Eigenvalues, SVD & Matrix Secrets

Eigenvectors and Eigenvalues

Singular Value Decomposition (SVD)

Low-Rank Approximation and LoRA

Check Your Understanding