Logits, temperature, and the token that comes out

The tower is built. Text became tokens, tokens became vectors, attention let them talk, and a full block reshaped them. At the very top sits one output vector - two numbers, in our toy. And a reader is waiting for a word. This chapter is the machine’s last mechanical step, the one chapter 1 promised: turn a vector into a bet over the whole vocabulary, then actually place the bet. It is also the chapter where our hand-built toy gets caught pretending to know English.

Observe the table read backwards

To score a word, the model computes a dot product between the context vector and - surprise - that word’s own embedding row from chapter 3. The same table, read backwards: rows that were lookups on the way in become scorers on the way out. Many production models genuinely share these weights; the trick is called weight tying. Each raw score is a logit; softmax - the accountant from chapter 4 - converts logits into shares of 100 %.

Predict the toy’s first word

The context is the course trace, “eat an”. Every number in this machine was written by hand for teaching, and none of it was ever trained.

Your bet

After “eat an”, which noun does our hand-built model actually bet on?

orange, like the corpus in chapter 1catIt refuses to answer

Nobody grades this. Commit, and let the instrument settle it.

Generation studio / from vector to token

The top of the tower produces a vector; the reader wants a word. Watch the vector become a bet, then let the decoding policy choose.

Temperature

1.0

Decoding policyGreedy (argmax)Top-kTop-p (nucleus)Full sampling

Adjust k and p

k (words kept)

3

p (cumulative mass)

0.80

From vector to bet

Current context → output vector h = [3.659, 1.933]

Word	Logit h · E[word]	Share (after T)	Final share
cat · #314kept	4.392	46.1 %	46.1 %
dog · #271kept	4.022	31.8 %	31.8 %
queen · #409kept	2.954	10.9 %	10.9 %
king · #408kept	2.907	10.4 %	10.4 %
bicycle · #734kept	-0.837	0.2 %	0.2 %
car · #733kept	-0.853	0.2 %	0.2 %
orange · #901kept	-1.638	0.1 %	0.1 %
apple · #612kept	-2.236	0.1 %	0.1 %

The loop, for real

Every draw reruns everything: causal block → logits → policy → token appended to the context.

eatan

Draw a tokenChange the diedie #2019

ResetCopy link

Manipulate the policy

There it is: cat, at roughly 46 % - and orange buried near the bottom at 0.1 %, with only apple faring worse. Hold that thought; it is the most instructive number in this course, and we will come back to it.

First, work the controls. Slide temperature down to 0.5: the winner’s share balloons. Slide it to 2: the distribution flattens, underdogs breathe. Note what never changes: the ranking. Then switch policies. Greedy always takes the maximum - press Draw a token a few times and watch the toy write “eat an cat cat cat…”, a real pathology called degeneration that plagued early text generation. Top-k and top-p cut the tail before drawing; full sampling draws from everything. The die makes each draw reproducible: same seed, same text, every time.

Explain the division of labor

The pipeline you just drove has a strict division of labor, and it is worth naming, because public debate constantly confuses the two halves:

• The model computes: context vector → logits → (with temperature and softmax) a probability for every word. That is where its “knowledge” lives.
• The decoding policy chooses: greedy, top-k, top-p, full sampling - these never touch the model. They decide how boldly to spend the probabilities the model produced.

Temperature belongs to the second half. Dividing all logits by the same number rescales confidence but preserves order - which is exactly what the second challenge mission demonstrates. (Chapter 4’s “score sharpness” slider was a teaching lens on attention; this is the real generation temperature.)

1. The tower outputsh = one vector
2. The table scoreslogit(word) = h · E[word]
3. The policy picksnext token → back into the context

Now the cat. Why does a machine with every mechanism in place - tokens, embeddings, attention, a full block, weight tying - bet on cat after “eat an”? Because mechanisms carry no knowledge. Every matrix in this toy was authored by hand to be legible, not to be right. The chapter 1 counting table, dumb as it was, at least learned from data; our beautiful tower has never seen any. Fluency of machinery and truth of output are entirely separate achievements.

Reflect on what temperature can never do

Checkpoint

Greedy decoding picks “cat” here at every temperature. Why can't temperature change that?

Greedy decoding picks “cat” here at every temperature. Why can't temperature change that?

ABecause greedy mode bypasses the temperature entirely and reads the chapter 1 counting table, whose corpus counts never change with T.BBecause temperature is switched off while greedy is selected.CDividing every logit by the same T preserves their order, and greedy takes the maximum.

Check answer

Connect the loop, then break the spell

The loop is closed. Text → tokens → vectors → attention → block → logits → probabilities → sampled token → back into the context: you have now touched every stage of the model map, and nothing in it is magic. But the toy’s confident “eat an cat” leaves exactly one mystery standing, and it is the big one: where do good numbers come from? Nobody hand-writes the matrices of a real model. The machine writes them itself, by being wrong millions of times and nudging every number to be slightly less wrong. That process - training - is the next chapter, and it ends with this same bet flipping to orange.

Sources and scope

• Press & Wolf (2017) shows that sharing the embedding and unembedding weights - the tying this lab makes visible - improves language models.
• Holtzman et al. (2020) documents degeneration under greedy and beam decoding and proposes nucleus (top-p) sampling.
• The vocabulary is the eight handcrafted chapter 3 rows; the prefix’s function words exist only as input fixtures. No number here comes from a trained model - that gap is the point of the chapter.
• Content and claims reviewed on 2026-07-18.