In this lesson
From raw continuer to assistant
A base model only continues text. Walk the same tower through its finishing school - exemplary dialogues, then human preferences - and learn what that steering does and does not guarantee.
By the end, you can
- Explain what supervised fine-tuning and preference tuning each change about a base model.
- State what alignment steering does not guarantee, and why tools and retrieval exist.
Before you start: Training, the machine that fixes its own numbers
Ask a freshly trained model why the sky is blue, and it may reply with more questions about the sea and the grass. It is not broken. It is doing, flawlessly, the only thing chapters 1 through 7 ever built: continuing text the way its training data continues. On the web, one question is most often followed by another question, a list, an ad. The distance between that magnificent parrot and the assistant that answers you politely every day is short, mechanical, and worth seeing exactly - because it is also where the last illusions about these systems live.
Observe one tower, three schools
The lab below shows the same architecture at three stages: the base model, the model after supervised fine-tuning (SFT), and the model after preference tuning. Same tower, same attention, same softmax - only the numbers differ. Every output shown is a hand-written illustration of behaviors documented in the research; no real model was queried.
Predict the base model’s reaction
Give the base stage a simple instruction: Translate “cat” into French.
Your bet
What does the base model do with a clear, polite instruction?
Nobody grades this. Commit, and let the instrument settle it.
One architecture, three sets of numbers. These outputs are hand-written illustrations of documented behaviors, not measurements from a real model.
What the model writes
Translate “cat” into French.
Translate “dog” into French. Translate “house” into French. Exercise 3: conjugate the following verbs.
Top candidates for the next token
| Token | Share |
|---|---|
| Translate | 38 % |
| Exercise | 21 % |
| chat | 9 % |
| …other tokens | 32 % |
Inspect the SFT mask
During SFT, the same chapter 7 cross-entropy runs - but only the response tokens count toward the loss. The model learns to answer, not to repeat the question.
Whyistheskyblue?Becauseairscattersbluelightmore.
prompt · ignored by the lossresponse · scored by the loss
Manipulate the three stages
The base model answered your instruction with… more homework. Nothing in next-token training rewards obeying; the web’s most likely continuation of an exercise is the next exercise. Now click through the stages and watch the lead token shift. With SFT, “chat” jumps to the top: thousands of exemplary prompt-and-answer dialogues taught the tower that after an instruction comes its execution. Then try the cruel-insult prompt across all three stages. The base model continues a forum thread. The SFT model helpfully complies - it was trained to answer, and it answers. Only the preference-tuned model declines and offers an alternative.
Challenge Two missions. One: open the SFT mask inspector and explain why the prompt tokens are excluded from the loss - what would a model trained on unmasked dialogues learn to do? Two: across the three stages of the insult prompt, track the probability of the refusal-starting token. Say in one sentence what kind of data moved it.
Explain the finishing school
Both stages of the finishing school reuse machinery you already own:
- Supervised fine-tuning is chapter 7’s cross-entropy on a curated diet: dialogues written to be exemplary, with the loss masked so only response tokens are scored. The model learns the shape of answering - it does not learn new facts about the sky.
- Preference tuning starts where “just imitate” runs out: raters compare two candidate responses, and an optimization (RLHF, DPO, and cousins) shifts probability toward the preferred kind. That is how “decline and offer an alternative” can beat “comply fluently” even though both are grammatical continuations.
- Chapters 1–7 builta continuer of text
- SFT teachesthe shape of answering
- Preferences teachwhich answers people choose
What none of this does is equally important. Steering shifts probabilities; it installs no truth-checker and no rule engine. The assistant still plays chapter 1’s game at every single token - which is why it can decline an insult and misstate a date in the same breath, and why real products bolt on retrieval (fetch documents and let the model read them in context) and tools (let it call a calculator or a search engine) instead of trusting frozen weights to know everything. And the raters’ tastes, blind spots, and biases flow directly into what “preferred” means.
Alignment is steering, not certification. A preference-tuned model produces the kind of text raters preferred - a moving, human, imperfect target that correlates with helpful and harmless without guaranteeing either. When you read a confident answer, chapter 6’s division of labor still applies: fluency is the model’s game; truth was never the scoring rule.
Reflect on what actually changed
Checkpoint
Between the base model and the assistant, what actually changed?
Connect the whole journey
Full circle. A model once fumbled counting the r’s in strawberry - you now know it never saw letters (chapter 2). Addresses became geometry you dragged by hand (3), context flowed between tokens through attention you traced score by score (4), a full block stacked position, mixture, and transformation (5), a vector became a bet and a token (6), the bet was trained from absurd to right in front of you (7), and the raw continuer just went through finishing school (8). The cat from chapter 2 can finally retire: every number between raw text and a polite answer now has a story you can tell - and an instrument you can reopen, reset, and share whenever the story needs checking.
Sources and scope
- Ouyang et al. (2022) describes the SFT-then-preference pipeline (InstructGPT) that this chapter’s three stages illustrate, including base-model instruction failures.
- Rafailov et al. (2023) presents Direct Preference Optimization, a widely used way to learn from chosen-versus-rejected response pairs.
- All continuations and token shares in this lab are authored teaching fixtures depicting documented behavior patterns; none are outputs measured from a real model.
- Content and claims reviewed on 2026-07-18.