Natural Language Processing · free preview
Masked Language Models and Encoders
Three transformer architectures, bidirectional attention without a mask, the cloze-style 80/10/10 masking recipe, and fine-tuning for classification and tagging.
1. Masked Language Models and Encoders
Masked Language Models and Encoders
2. Three architectures, three jobs
Which pairing of architecture to task is right?
3. Expected masks under 80/10/10
Across a large corpus, tokens per window are selected for manipulation under the BERT recipe, 80 percent masked, 10 percent random, 10 percent kept. In expectation, how many of them become [MASK]? Answer with the number (one decimal is fine).
4. Why the 10 percent branches
Why does the recipe replace 10 percent of selected tokens with random words and leave 10 percent unchanged, instead of always inserting [MASK]?
5. Counting BIO tags
A named entity recognizer uses BIO tagging over entity types. How many distinct tags does the per-token classifier need? Answer with the number.
6. MLM loss at a masked slot
At one selected slot the model assigns the true token probability . Compute the MLM loss at that slot in nats, rounding to three decimals.
Practice this interactively
Free account · instant grading · spaced review that schedules itself.
Start this course for free