A vector space is the set of all possible vectors of a given dimension. ℝ² is the 2D plane — every point (x, y) is a vector. ℝ⁷⁶⁸ is the 768-dimensional space where transformer embeddings live.
What makes something a vector space? Two properties:
That's it. These two properties mean you can navigate the space freely — any linear combination of vectors stays in the space.
A basis is a set of vectors that lets you describe any vector in the space as a unique linear combination. In ℝ², the standard basis is:
e₁ = [1, 0] e₂ = [0, 1]
Any vector [x, y] = x·e₁ + y·e₂. The coordinates (x, y) are relative to this basis. Choose a different basis, and the same point gets different coordinates.
An embedding maps discrete items (words, tokens) to vectors in a continuous space. The embedding layer in a transformer is literally a matrix E of shape (vocab_size × dmodel). Token i → row i of E.
During training, the entries of E are adjusted so that:
The embedding space learns a basis where proximity equals semantic similarity. The dimensions aren't pre-labeled ("dimension 37 = royalty"). They emerge from training as the model discovers what representations make the prediction task easier.
Vectors alone don't encode order. [0.5, −0.2, …] doesn't say "I'm the third word." The transformer solves this by adding a position-dependent vector to each token embedding:
input_vector = embedding(token) + position_encoding(position)
The position encoding uses sine and cosine functions at different frequencies, creating a unique pattern for each position. This addition is why transformers can process all tokens in parallel — position information is baked into the vectors themselves, not into the order of processing.
Q1: What does the embedding layer of a transformer actually do mathematically?
A. It runs a neural network on each token B. It looks up the token's ID in a matrix and returns the corresponding row as a vector C. It counts how many times each token appears D. It applies the attention formula to each token individuallyQ2: Why does the transformer add positional encodings instead of processing tokens sequentially?
A. Sequential processing would be too slow — adding position vectors lets all tokens be processed in parallel B. The model can't do addition, so this is the only option C. Position is irrelevant — the encoding is just a historical artifact D. It's required by the GPU hardwareQ3: The embedding matrix E has shape (50,257 × 768) for a typical GPT model. What does the number 50,257 represent?
A. The maximum context length in tokens B. The vocabulary size — the number of distinct tokens the model knows C. The number of attention heads D. The number of transformer layers