The Transformer Block

The Transformer Block

Stacking, packing, caching

One block in NumPy

A 3-dimensional token vector is $[4, 2, 5]$. Compute the first component after layer normalization (no gain or offset), rounding to three decimals.

A transformer block has model dimension $d = 1024$ and feedforward hidden width $d_{ff} = 4d = 4096$. Counting the two feedforward matrices and the four attention projections (at head width $d$), how many times more parameters does the feedforward layer hold than the attention projections? Answer as a decimal.

Which statement about the components of a transformer block is correct?

Ignoring biases, embeddings, and the head, a stack of $L = 96$ blocks at model dimension $d = 1024$ holds about $12Ld^2$ parameters. Compute this estimate and answer in scientific notation like 1.2e9 or as a plain integer.

During autoregressive generation without any cache, each new token triggers recomputation of key and value projections. What does the KV cache do instead, and what does it cost?