Natural Language Processing · 免费预览
掩码语言模型与编码器
三种 Transformer 架构：无掩码的双向注意力、填空式 80/10/10 掩码方案，以及用于分类和标注的微调。
1. Masked Language Models and Encoders
Masked Language Models and Encoders
2. Three architectures, three jobs
Which pairing of architecture to task is right?
3. Expected masks under 80/10/10
Across a large corpus, tokens per window are selected for manipulation under the BERT recipe, 80 percent masked, 10 percent random, 10 percent kept. In expectation, how many of them become [MASK]? Answer with the number (one decimal is fine).
4. Why the 10 percent branches
Why does the recipe replace 10 percent of selected tokens with random words and leave 10 percent unchanged, instead of always inserting [MASK]?
5. Counting BIO tags
A named entity recognizer uses BIO tagging over entity types. How many distinct tags does the per-token classifier need? Answer with the number.
6. MLM loss at a masked slot
At one selected slot the model assigns the true token probability . Compute the MLM loss at that slot in nats, rounding to three decimals.