Natural Language Processing · free preview
Contextual Embeddings and Interpretability
Contextual embeddings as per-instance meaning vectors, nearest-neighbor word sense disambiguation, anisotropy and z-scoring, probing classifiers, and the logit lens.
1. Contextual Embeddings and Interpretability
Contextual Embeddings and Interpretability
2. Type or instance?
What is the difference between the vector word2vec assigns a word and the vector a BERT-style encoder assigns it?
3. Averaging into a sense embedding
Two labeled instances of the water sense of a word have 2-d embeddings and . Compute their sense embedding by averaging, and give the first component, rounding to two decimals.
4. The anisotropy problem
Raw final-layer contextual embeddings give cosines near 1 for random word pairs. What is this property called, what causes it, and what is the standard fix?
5. Reading a probe result
A linear probe recovers part of speech from layer 4 activations with 95 percent accuracy. Which conclusion is justified?
6. Through the unembedding, early
What does the logit lens do, and why does the trick only approximately work?
Practice this interactively
Free account · instant grading · spaced review that schedules itself.
Start this course for free