Decoding and Sampling Strategies

Decoding and Sampling Strategies

Temperature and truncation

Four decoders in NumPy

A model's next-token distribution is fog $0.33$, tide $0.22$, gulls $0.21$, horn $0.24$. Which word does greedy decoding emit? Answer with the word.

A decoder divides logits by $\tau = 4$ before the softmax. What happens to the probability of the leading token, and what generation quality changes with it?

A distribution over four words is $[0.5, 0.3, 0.15, 0.05]$. Top-2 sampling keeps the $2$ largest, discards the rest, and renormalizes. Compute the new probability of the top word, rounding to three decimals.

Two contexts both decode with top-p at 0.6. In context A the top two tokens hold 0.85 of the mass together, in context B the top five hold 0.55. What does the nucleus keep in each?

Which task and decoder pairing is most defensible?