Natural Language Processing · free preview
Text to Speech and Audio Codecs
Synthesis as language modeling over discrete audio tokens, codecs like ENCODEC with residual vector quantization, two-stage generation in VALL-E, and evaluation.
1. Text to Speech and Audio Codecs
Text to Speech and Audio Codecs
2. Codecs and residual vector quantization
Codecs and residual vector quantization
3. Residual quantization in miniature
Residual quantization in miniature
4. Two-stage generation and evaluation
Two-stage generation and evaluation
5. What a zero-shot synthesizer takes
A zero-shot TTS system must speak a novel text in a voice never heard in training. What does it receive as input at inference time?
6. Counting codec frames
The ENCODEC encoder emits embeddings at 75 Hz, one per frame. How many embedding frames does a -second clip of speech produce?
7. Counting the bits
A residual vector quantizer emits 8 codes per frame at 75 frames per second, and each code indexes a 1,024-entry codebook, costing 10 bits. How many bits per second of audio does the token stream carry?
8. Why two stages
VALL-E generates the first quantizer's codes with an autoregressive model and the remaining seven quantizers with non-autoregressive passes. What motivates the split?
9. Gradients through a lookup
Nearest-codeword assignment in vector quantization is not differentiable, which blocks end-to-end codec training. How does the straight-through estimator resolve this?
Practice this interactively
Free account · instant grading · spaced review that schedules itself.
Start this course for free