Natural Language Processing · free preview

Pretraining and Perplexity

Next-token pretraining with teacher forcing, cross-entropy per position, parallel training on packed windows, and perplexity as exp of mean loss.

1. Pretraining and Perplexity

2. Perplexity and its caveats

Perplexity and its caveats

3. Teacher-forced loss and perplexity

Teacher-forced loss and perplexity

4. One position's loss

At one position the model assigns the correct next token probability $0.05$ . Compute that position's cross-entropy loss in nats, rounding to three decimals.

5. Perplexity is exp of the loss

A model's mean cross-entropy on a held-out text is $3$ nats per token. Compute its perplexity, rounding to two decimals.

6. What history does training feed?

During pretraining at position t, the model predicts token $w_{t+1}$ . What exactly conditions the prediction, and what is this scheme called?

7. Every position is an example

A training window holds $N = 1,024$ tokens. Thanks to the causal mask, how many next-token training examples does one forward pass over this window provide? Answer with the number.

8. Comparing perplexities across models

Model A reports perplexity 12 on a benchmark, model B reports 15, but the two use different tokenizers. What can be concluded?

Practice this interactively

Free account · instant grading · spaced review that schedules itself.

Start this course for free