Natural Language Processing · 免费预览
基于字节对编码的子词分词
为什么词表会介于词与字符之间，Unicode 文本如何变为字节，以及字节对编码如何学习合并并对新词编码。
1. Subword Tokenization with Byte-Pair Encoding
Subword Tokenization with Byte-Pair Encoding
2. BPE learns merges from counted pairs
BPE learns merges from counted pairs
3. Training and applying a BPE tokenizer
Training and applying a BPE tokenizer
4. The first BPE merge
A BPE trainer runs on this word-frequency corpus (each word ends with the end-of-word marker _): hared (x2), hars (x4), tideed (x2), tides (x6).
Which adjacent symbol pair is merged first?
5. UTF-8 byte length
How many bytes does UTF-8 use to encode the single character 日?
6. Effect of the first merge
Starting from the same corpus (hared (x2), hars (x4), tideed (x2), tides (x6), with the end-of-word marker _), the trainer performs the first merge s+_. After this merge, across all words in the corpus, in how many token positions does the new symbol s_ occur? (Weight each word by its frequency.)
7. Encoding after early merges
BPE is trained on (hared (x2), hars (x4), tideed (x2), tides (x6), end-of-word marker _) for exactly 5 merges, producing the ordered merge list ['s+_', 't+i', 'ti+d', 'tid+e', 'h+a'].
Encoding replays the merges in order. Write the resulting tokenization of the word "tides" as symbols joined with + (include the end-of-word marker _ as its own symbol if it stands alone).
8. Encoding an unseen word
With the same trained merge list ['s+_', 't+i', 'ti+d', 'tid+e', 'h+a'], encode the word "harnt", which never appeared in the training corpus.