Self-Attention

Self-Attention

Queries, keys, values, and heads

Attention weights in NumPy

A token attends to three prior tokens with raw scores $2$, $1$, $-1$. Compute the attention weight for the first token after the softmax, round to three decimals.

A query-key dot product is $12$ with head dimension $d_k = 25$. Compute the scaled score $\frac{\mathbf{q} \cdot \mathbf{k}}{\sqrt{d_k}}$ and round to three decimals.

Match each projection to its role in one attention head.

In the masked attention matrix for a causal language model, future positions receive the value $-\infty$ before the softmax rather than 0. Why?

A transformer has model dimension $d = 512$ and $A = 16$ attention heads, with the usual choice $d_v = d/A$ so all heads share the budget. What is $d_v$? Answer with the number.