Subword Tokenization with Byte-Pair Encoding

Subword Tokenization with Byte-Pair Encoding

BPE learns merges from counted pairs

Training and applying a BPE tokenizer

A BPE trainer runs on this word-frequency corpus (each word ends with the end-of-word marker _): hared (x2), hars (x4), tideed (x2), tides (x6).

Which adjacent symbol pair is merged first?

How many bytes does UTF-8 use to encode the single character 日?

Starting from the same corpus (hared (x2), hars (x4), tideed (x2), tides (x6), with the end-of-word marker _), the trainer performs the first merge s+_. After this merge, across all words in the corpus, in how many token positions does the new symbol s_ occur? (Weight each word by its frequency.)

BPE is trained on (hared (x2), hars (x4), tideed (x2), tides (x6), end-of-word marker _) for exactly 5 merges, producing the ordered merge list ['s+_', 't+i', 'ti+d', 'tid+e', 'h+a'].

Encoding replays the merges in order. Write the resulting tokenization of the word "tides" as symbols joined with + (include the end-of-word marker _ as its own symbol if it stands alone).

With the same trained merge list ['s+_', 't+i', 'ti+d', 'tid+e', 'h+a'], encode the word "harnt", which never appeared in the training corpus.