In this lesson
Training, the machine that fixes its own numbers
Loss, gradient, small step, repeat. Free exactly two numbers, watch gradient descent drag a row of the map, and see chapter 6's absurd bet flip.
By the end, you can
- Explain training as loss measurement, gradient computation, and repeated small parameter updates.
- Explain why learned geometry serves prediction rather than human intuition.
Before you start: Logits, temperature, and the token that comes out
Nobody wrote the numbers in a real language model. Not the embedding table, not the Q, K, V projections, not the MLP - not one of the billions. Chapter 6 ended with the proof that this matters: our lovingly hand-built tower, given “eat an”, bet 46 % on cat. This chapter shows the mechanism that writes the numbers: measure how wrong you are, compute which direction makes you less wrong, take a small step, repeat. By the end, the same bet lands on orange - and you will have watched every step of the repair.
Observe what gets to move
Real training adjusts every parameter at once, which is impossible to watch. So this lab freezes the entire tower and frees exactly two numbers: the coordinates of orange’s row in the chapter 3 embedding table - the very row you once dragged by hand. The context vector h for “eat an” stays fixed; the only question is where the machine will drag that one row, and it answers with three moves: a loss that scores the current bet, a gradient that points uphill, and a small step the other way.
Predict the shape of learning
The loss starts at 6.8 - the price of giving the right answer a 0.1 % chance.
Your bet
At learning rate 0.1, what will the loss curve look like as you train?
Nobody grades this. Commit, and let the instrument settle it.
The whole tower is frozen except two numbers: orange’s row in the chapter 3 table. Watch gradient descent drag it.
The model’s bet
| Word | Share |
|---|---|
| cat | 46.1 % |
| dog | 31.8 % |
| queen | 10.9 % |
- Loss −log p(target)
- 6.804
- p(target)
- 0.1 %
The model still bets wrong. Keep training.
Loss curve
0 → 6.804
The map, during training
Seven rows stay fixed; orange’s row (in orange) is dragged step by step. The trail shows its path.
Inspect the gradient
| Context vector h (frozen) | [3.659, 1.933] |
|---|---|
| Pull strength (1 − p) | 0.999 |
| Current gradient | [-3.655, -1.931] |
| Current orange row | [-0.680, 0.440] |
gradient = −(1 − p(target)) × h; update: row ← row − rate × gradient
Manipulate the descent
Train step by step and watch three displays move together: the loss falls, p(orange) climbs, and on the map, the orange row physically slides - gradient descent is a drag operation, executed by calculus. At rate 0.1 the bet flips at step 4. Now reset to rate 0.01: same destination, thirty-eight steps of crawling. Rate 1 crosses in a single stride. In this two-parameter bowl, bigger is simply faster; hold that thought for the aside below, because it is not true at scale.
Challenge Two missions. One: after the bet flips, keep training and watch the pull strength (1 − p) in the inspector. Explain why training slows itself down as the model gets it right. Two: look at where the orange row ends up on the map - right next to cat. Check the neighbors it left behind. Something about the chapter 3 map has been quietly sacrificed; name it.
Explain the three-step loop
The loss is cross-entropy: −log p(target). It is brutal on confident mistakes - betting 0.1 % on the truth costs about 7, betting 90 % costs 0.1 - which is exactly the pressure that makes a model calibrate.
The gradient, for this softmax head, has a closed form worth reading aloud:
gradient = −(1 − p(target)) × h
Pull the target’s row toward the context vector, with a force proportional to how surprised you were. When p(target) is tiny, the pull is fierce; as the model gets it right, the pull fades to nothing. Training is self-extinguishing - the first challenge mission shows it live. And the update is just row ← row − rate × gradient, the whole ritual of modern AI in one line.
- Measureloss = −log p(orange)
- Differentiategradient = −(1 − p) × h
- Step and repeatrow ← row − rate × gradient
Now scale the picture up honestly. A real model computes this same gradient for every parameter simultaneously - embedding rows, Q, K, V, MLP weights, all of them - using backpropagation, which is the chain rule applied through the whole tower. It does so on billions of sentences, not one. And your second challenge found the cost of our shortcut: to win one bet, the machine dragged orange into cat’s neighborhood - their cosine went from −0.27 to 0.998. The map now serves prediction and horrifies a lexicographer. That is the deepest lesson this toy can teach: learned geometry answers to the loss, not to human taxonomy. Real models balance billions of sentences, so their geometry ends up useful for many bets at once - but it answers to the same master.
What the bowl cannot show. Two free parameters make a convex bowl, where a giant learning rate is simply efficient. A billion-parameter loss landscape is nothing like a bowl, and oversized steps genuinely destroy training runs there - that claim rests on the optimization literature, not on this lab. Also: training inserted no fact and no understanding. It made one number go down. Grammar, arithmetic, “knowledge,” and the biases of the data all enter a real model the same way: as whatever helps that number go down.
Reflect on what the gradient says
Checkpoint
The gradient here is −(1 − p(target)) × h. What is the machine literally instructed to do?
Connect the repaired machine to the real ones
The toy now completes “eat an” with orange. Every stage of the model map has been built, and the one mystery chapter 6 left standing - where good numbers come from - is now a mechanism too: wrongness, measured; direction, computed; steps, repeated. Scale this to billions of parameters and oceans of text and you get what practitioners call a base model: a magnificent continuer of text, and nothing else. It completes; it does not answer, refuse, or follow instructions. The short distance between that raw continuer and the assistant you talk to every day is the last chapter of this course.
Sources and scope
- Rumelhart, Hinton & Williams (1986) introduces backpropagation - the chain-rule machinery that computes these same gradients through every layer of a real network.
- Bengio, Ducharme, Vincent & Jauvin (2003) trains embedding rows exactly this way - by gradient descent on next-token cross-entropy - at real scale.
- This lab differentiates two free parameters analytically; no backpropagation through the tower is performed, and the convex-bowl behavior of the learning rates does not generalize to real loss landscapes.
- Content and claims reviewed on 2026-07-18.