In this lesson
The Transformer block, the floor a model stacks
One complete floor: positions, normalization, attention, MLP, and the residual stream - plus an exact experiment on what the block does with word order.
By the end, you can
- Trace one vector through positions, normalization, attention, the MLP, and both residual additions.
- Explain why a Transformer needs position information at all.
Before you start: Attention, where tokens start talking
Dog bites man. Man bites dog. Same three words; only one of them is news. Now recall what you built in chapter 4: attention compares queries with keys through dot products, and nothing - literally nothing - in a dot product knows where a token sits. So what does the mechanism that lets tokens talk actually do with word order? Before stacking it into a real model, that question deserves an exact answer - and, if the answer alarms you, a fix.
Observe the machine’s blind spot
A Transformer is a tower of identical floors, and this chapter walks one full floor: add position vectors, normalize, attend, add the result back, normalize again, transform each token with a small neural network, add that back too. The instrument below runs the same word through the same floor twice - once in the original order, once reversed - so any difference between the two runs must come from order alone.
Predict the effect of scrambling
Keep Position vectors off. The tracked token orange sits at the end of “eat an orange” and at the start of the reversed phrase.
Your bet
Positions off: reversing the word order does what to orange's block output?
Nobody grades this. Commit, and let the instrument settle it.
The same word crosses the same block in two sentence orders. Toggle the position vectors and measure what the block can - and cannot - see.
Original order
eatanorange
- position
- p2
- vector added
- none
- block output
- [-1.197, 1.385]
Reversed order
orangeaneat
- position
- p0
- vector added
- none
- block output
- [-1.197, 1.385]
Identical outputs: without positions, the block treats the sentence as a bag of words.
Inspect the full floor (original order)
| Stage | Vector |
|---|---|
| Input embedding | [-0.680, 0.440] |
| + position vector | [-0.680, 0.440] |
| RMSNorm (before attention) | [-1.187, 0.768] |
| Attention mixture | [-0.517, 0.243] |
| + residual (stream + attention) | [-1.197, 0.683] |
| RMSNorm (before MLP) | [-1.228, 0.701] |
| MLP (2 → 3 → 2, ReLU) | [0.000, 0.701] |
| + residual = block output | [-1.197, 1.385] |
| Q ↓ / K → | eat | an | orange |
|---|---|---|---|
| eat | 0.482 | 0.482 | 0.036 |
| an | 0.482 | 0.482 | 0.036 |
| orange | 0.065 | 0.065 | 0.871 |
Each row sums to 1. A real decoder keeps the causal mask from chapter 4; it is removed here so the order experiment is exact.
Manipulate the floor
If you bet on “identical,” you have just verified something most explanations skip: the whole floor - attention, normalization, MLP, residuals - is permutation-equivariant. Scramble the sentence and each word’s output follows it around unchanged, to the last digit.
Now switch Position vectors on. Each slot adds its own small vector (p0, p1, p2) to whatever token sits there, before anything else happens. The two runs immediately disagree: order has entered the computation as geometry. Open the inspector and follow the eight stages; notice the MLP’s output for the target - its first coordinate is exactly 0.000, clipped by the ReLU, the max(0, z) gate defined below - a small honest scar of how these networks compute.
Challenge Two missions. One: with positions on, track the middle word an. Reversing a three-word phrase leaves the middle in place, so it receives the same position vector in both runs - yet its output still changes. Explain where the difference sneaks in. Two: in the weights matrix, every row sums to exactly 1. Say which chapter-4 ingredient guarantees that.
Explain the floor plan
Here is the floor in order, and why each part earns its place:
- Position vectors stamp “you are in slot 2” into the token’s geometry, because nothing downstream can see order otherwise.
- RMSNorm rescales each vector to a standard size before it enters a sublayer - after many floors, unchecked values would drift huge or tiny. It rescales without recentering: direction survives.
- Attention is the only step where positions exchange information; you built it in chapter 4.
- The MLP (a small multi-layer perceptron) is its complement: it never looks sideways. Each token, alone, is expanded (2 → 3 here), passed through a ReLU that keeps only positive evidence, and contracted back (3 → 2). Real models expand about four-fold, with thousands of dimensions.
- Residual additions are the quiet masterstroke. The block does not replace the token’s vector; it amends it:
output = input + correction, twice per floor. That running total is called the residual stream, and it is why dozens of floors can stack without the signal dissolving.
- Chapter 4 builtattention: tokens talk
- This floor addspositions · norm · MLP · residuals
- A real model stacksN × this floor
The same floor, in formulas
RMSNorm rescales a vector x to x / √(mean(x²)), so its root-mean-square becomes 1 while its direction is preserved (real models then multiply by a small learned gain).
The MLP computes W₂ · ReLU(W₁x + b₁) + b₂, with ReLU(z) = max(0, z).
One pre-norm floor, written as amendments to the residual stream h:
h ← h + Attention(RMSNorm(h))
h ← h + MLP(RMSNorm(h))
This lab adds hand-authored position vectors. Production models learn their position encodings or rotate Q and K by position (RoPE); the reason they exist is the same.
No single part is “where the model thinks.” Attention moves information between positions; the MLP transforms it in place; they alternate, floor after floor. And a real decoder keeps the causal mask from chapter 4 inside every floor - this lab removes it only so the order experiment is mathematically exact.
Reflect on the bag of words
Checkpoint
With positions off (and no mask), reversing the sentence leaves every word's output unchanged. Why?
Connect the floor to the tower
The course trace has now crossed a complete floor: the target entered as the chapter-3 lookup [−0.68, 0.44] and left taller, reshaped by neighbors, positions, and the MLP. A real model repeats this floor dozens of times, with many attention heads per floor, before anything is said out loud. Which raises the final mechanical question: at the top of the tower sits a vector - and a reader wants a word. Turning that vector into a bet over the whole vocabulary, and then into a chosen token, is the next chapter.
Sources and scope
- Vaswani et al. (2017) defines the Transformer block: attention plus a position-wise feed-forward network, residual connections, and normalization.
- Zhang & Sennrich (2019) introduces RMSNorm, the recenter-free normalization used in this lab and in many modern models.
- Su et al. (2021) presents rotary position embeddings (RoPE), the production alternative to the additive position vectors shown here.
- The floor is one handcrafted 2D block with a 3-unit MLP; attention runs unmasked in this lab so the permutation experiment is exact.
- Content and claims reviewed on 2026-07-18.