Natural Language Processing · 免费预览

上下文嵌入与可解释性

上下文嵌入作为逐实例含义向量、最近邻词义消歧、各向异性与 z 分数标准化、探测分类器，以及 logit 透镜。

1. Contextual Embeddings and Interpretability

Contextual Embeddings and Interpretability

2. Type or instance?

What is the difference between the vector word2vec assigns a word and the vector a BERT-style encoder assigns it?

3. Averaging into a sense embedding

Two labeled instances of the water sense of a word have 2-d embeddings $[1.04, 0.01]$ and $[0.77, 0.44]$ . Compute their sense embedding by averaging, and give the first component, rounding to two decimals.

4. The anisotropy problem

Raw final-layer contextual embeddings give cosines near 1 for random word pairs. What is this property called, what causes it, and what is the standard fix?

5. Reading a probe result

A linear probe recovers part of speech from layer 4 activations with 95 percent accuracy. Which conclusion is justified?

6. Through the unembedding, early

What does the logit lens do, and why does the trick only approximately work?

进入交互式练习

免费注册 · 即时批改 · 自动安排间隔复习。

免费开始这门课