Lesson card · Tokens & next-token prediction
How a model learns
- Pre-training
- Instruction tuning
- Preference tuning
- Assistant
Three stages, from text predictor to assistant
Stage 1, pre-training: the model reads a huge pile of text and practises predicting the next token, billions of times.
Stage 2, instruction tuning: it trains on smaller sets of good question-and-answer examples, so it behaves like an assistant.
Stage 3, preference tuning: people rank answers, and the model is nudged toward the ones they prefer.
Most of the knowledge comes from stage 1. Most of the manners come from stages 2 and 3.
New words pre-training: the first, biggest training stage: predicting the next token across a huge amount of text
New words instruction tuning: extra training on example questions and good answers so the model follows instructions
New words preference tuning: training that nudges the model toward answers people ranked higher