预训练与困惑度

Pretraining and Perplexity

Perplexity and its caveats

Teacher-forced loss and perplexity

At one position the model assigns the correct next token probability $0.05$. Compute that position's cross-entropy loss in nats, rounding to three decimals.

A model's mean cross-entropy on a held-out text is $3$ nats per token. Compute its perplexity, rounding to two decimals.

During pretraining at position t, the model predicts token $w_{t+1}$. What exactly conditions the prediction, and what is this scheme called?

A training window holds $N = 1,024$ tokens. Thanks to the causal mask, how many next-token training examples does one forward pass over this window provide? Answer with the number.

Model A reports perplexity 12 on a benchmark, model B reports 15, but the two use different tokenizers. What can be concluded?