大作业：一个小型语言模型

Capstone: A Mini Language Model

A tokenizer from scratch

Counts, smoothing, perplexity

Reading the sample, bridging to the network

Byte-pair encoding trains on a corpus whose words are split into characters. After eight merge steps the vocabulary contains pieces like er and ing. What decides the very first merge?

A tokenizer starts from $29$ distinct symbols, characters plus the word-end marker. After $9$ merge steps, each adding exactly one new symbol to the vocabulary, how many distinct subword symbols exist?

In a subword model with vocabulary size $V = 32$, the token ferry occurs $6$ times and the pair the ferry occurs $1$ times. With add-k smoothing at $k = 0.1$, what is $P(\text{ferry} \mid \text{the})$, to three decimal places?

In the capstone, the raw trigram counts are far sparser than the bigram counts, yet the trigram system posts the lower held-out perplexity. What property produced that result?

A sample from the capstone model contains the strings ferring and hourossing, words that never appear in the training corpus. What does their presence demonstrate about the setup?