What the model writes at this point
Its most likely continuation of a fixed prompt, computed from the saved checkpoint.
Loss during training
Training loss is the average over the previous 50 steps, so early on it lags behind the validation loss, which is measured at that exact step. Click the chart to jump to a checkpoint.
Do induction heads appear?
An induction head looks back to what followed the previous occurrence of the current token, which lets a model continue a repeated pattern. Scores are measured on random tokens repeated twice: 1 is a perfect induction (or previous-token) head.
All heads at this checkpoint
Rows are layers, columns are heads. Click a head to see its attention.
Each row is a token and shows where it looks (darker = more attention). Outlined cells are where an induction head would look: the token after the earlier copy of the current one.
What the model writes at this checkpoint
Greedy continuations of three fixed prompts, computed from the saved checkpoint by
python -m glassbox.induction (the browser shows them; it does not run old checkpoints).