In this lesson
Embeddings: meaning becomes geometry
Each identifier opens a locker holding a learned vector. Drag words around a toy map and measure what resembles what.
By the end, you can
- Explain how an embedding table turns an identifier into a vector.
- Interpret cosine similarity without mistaking it for a measure of truth.
Before you start: Tokenization, or how a model reads
The previous lesson ended with three addresses: #120, #18, #901. An address can distinguish orange from cat, but its size carries no meaning: token #901 is not “more orange” than token #314. To compute with useful relationships, the model needs something richer than a locker number. First, play with the geometry it will use.
Observe a vocabulary become a map
Imagine a small map where cat and dog begin in similar directions, while apple points elsewhere. This is not yet a map learned by a model; it is a hand-built scene for isolating the geometry.
Predict the cat’s big move
The instrument’s ranking scores how much every word resembles the observed token. Now suppose you grab cat and drag it all the way over to apple, on the far side of the map.
Your bet
If cat moves in with apple, what happens to its similarity with dog?
Nobody grades this. Commit, and let the instrument settle it.
This small map is handcrafted. Grab a point and drag it (sliders and arrow keys work too), and watch the cosine ranking follow.
| Token | ID | x | y | cosine |
|---|---|---|---|---|
| #314 | 0.82 | 0.72 | 1.00 | |
| #271 | 0.74 | 0.68 | 1.00 | |
| #408 | 0.34 | 0.86 | 0.89 | |
| #409 | 0.30 | 0.96 | 0.85 | |
| #901 | -0.68 | 0.44 | -0.27 | |
| #612 | -0.78 | 0.32 | -0.44 | |
| #733 | 0.20 | -0.82 | -0.46 | |
| #734 | 0.12 | -0.66 | -0.51 |
Teaching toy: these eight points did not come from a trained model. A real embedding is learned and often has hundreds of dimensions.
Manipulate directions
Grab a point and drag it; the map and table update together. Watch the ranking rather than the screen distance: cosine cares about direction from the origin, not absolute position.
Challenge Make car more similar to queen than to bicycle, in as few moves as you can. Then break something: find a single-point move that flips a cosine from positive to negative. The reset button forgives everything.
These eight points are handcrafted, not projected from a trained model. Real embedding tables are learned and often have hundreds or thousands of dimensions. Their axes do not arrive with human labels such as “animal” or “royal”; useful regularities are distributed across many directions.
Explain the machinery of the table
In 2013, researchers showed that some learned word-vector offsets captured recurring relationships. The famous shorthand was king − man + woman ≈ queen. It is a striking example, not an algebraic law: the result depends on the model and its training data.
- Token identifierorange · #901
- Table lookupE[901]
- Starting vector[−0.68, 0.44]
An embedding table is a matrix E with one row per token. For identifier i, E[i] returns vector eᵢ. During training, the values in E are adjusted whenever doing so helps the model predict the next token.
To compare two vectors, the instrument uses cosine similarity: a score that cares only about direction, never about length. A value near 1 means “similar directions” in this learned space; it does not mean “identical words” or “true statement.” At [0, 0], direction does not exist, so cosine similarity is undefined - the lab says so instead of inventing a zero.
The same score, in formulas
cos(a, b) = (a · b) / (||a|| × ||b||)
The dot product a · b grows when directions align; dividing by both lengths removes scale, so only the angle survives.
Context is still missing. For the same orange token identifier, the table returns the same row in “eat an orange” and “paint it orange.” The initial embedding carries learned regularities, including imperfections and biases from training; it does not yet express the precise role of this occurrence. Keep those two phrases in mind: they open chapter 4.
Reflect on what a cosine says
Checkpoint
Rows E[271] and E[314] have different lengths but point in nearly the same direction. What can you conclude?
Connect the fixed vector to its sentence
We now have one starting vector per token, but each lookup blissfully ignores the surrounding sentence: the vector for orange has no idea whether it is fruit or paint. In the next lesson, that same initial vector enters two contexts and leaves with two different outputs. Time to let the tokens talk to each other.
Sources and scope
- Mikolov, Yih & Zweig (2013) reports recurring syntactic and semantic offsets in learned word-vector spaces.
- The lab’s eight 2D points are handcrafted to teach lookup and cosine geometry; they are not measurements from a trained language model.
- Content and claims reviewed on 2026-07-18.