Words, Corpora, and Word Frequency

Words, Corpora, and Word Frequency

Word frequencies follow two empirical laws

Counting a corpus in a dozen lines

Count word tokens and word types in this sentence (split on spaces, ignore punctuation): "the ferry waits, the mast waits, and the ferry waits again."

A toy corpus contains these 23 tokens:

"pilot mast harbour dusk wake wake mast pilot dusk sail harbour pilot mast pilot wake wake sail pilot harbour wake sail dusk mast"

Compute the type-token ratio, rounded to three decimal places.

In a large corpus the most frequent type occurs 22,000 times. Zipf's law predicts the frequency of the type at rank 4 as $f(r) = k/r$. How many occurrences does it predict for rank 4?

The three most frequent types in a corpus occur 3,120, 1,560, and 520 times, and the long tail of remaining types accounts for 3,000 tokens. What fraction of all tokens does the single most frequent type account for? Round to three decimal places.

A corpus of 600,000 tokens has a vocabulary of 50,000 types. Assume Heaps' law $|V| = k\,n^{\beta}$ with $\beta = 0.56$. Predict the vocabulary size at 1,200,000 tokens.