Natural Language Processing · free preview

Dense Embeddings with Skip-Gram

Short dense vectors trained by binary classification on context pairs, negative sampling from a smoothed unigram distribution, and the analogy and bias properties of embedding spaces.

1. Dense Embeddings with Skip-Gram

2. Negative sampling and its loss

Negative sampling and its loss

3. Training skip-gram in NumPy

Training skip-gram in NumPy

4. Skip-gram probability from a dot product

A target and context embedding have dot product $\mathbf{c} \cdot \mathbf{w} = -2$ . Compute the classifier's probability $P(+ \mid w, c) = \sigma(\mathbf{c} \cdot \mathbf{w})$ and round to three decimals.

5. Smoothing the noise distribution

A two-word lexicon has unigram probabilities $0.99$ and $0.01$ . Compute the smoothed probability of the frequent word under $P_{0.75}$ , that is $\frac{0.99^{0.75}}{0.99^{0.75} + 0.01^{0.75}}$ and round to three decimals.

6. Skip-gram loss for one pair

A positive pair has $\mathbf{c}_{\text{pos}} \cdot \mathbf{w} = 1$ and each of the $2$ noise words has $\mathbf{c} \cdot \mathbf{w} = -1$ . Compute the loss $L = -[\log \sigma(1) + 2 \log \sigma(1)]$ and round to three decimals.

7. Which way does the target move?

A training pair has label $y = 1$ and the classifier currently outputs $\sigma(\mathbf{c} \cdot \mathbf{w}) = 0.7$ . According to the update $\mathbf{{w}} \leftarrow \mathbf{{w}} - \eta [\sigma(\mathbf{{c}} \cdot \mathbf{{w}}) - y]\,\mathbf{{c}}$ , what happens to the dot product $\mathbf{{c}} \cdot \mathbf{{w}}$ after the step?

8. Dense versus sparse, and structure

Which set of statements about static embeddings is correct?

Practice this interactively

Free account · instant grading · spaced review that schedules itself.

Start this course for free