Natural Language Processing · 免费预览
文本转语音与音频编解码器
将合成视为对离散音频词元的语言建模，如采用残差向量量化的 ENCODEC 等编解码器，VALL-E 中的两阶段生成，以及评估。
1. Text to Speech and Audio Codecs
Text to Speech and Audio Codecs
2. Codecs and residual vector quantization
Codecs and residual vector quantization
3. Residual quantization in miniature
Residual quantization in miniature
4. Two-stage generation and evaluation
Two-stage generation and evaluation
5. What a zero-shot synthesizer takes
A zero-shot TTS system must speak a novel text in a voice never heard in training. What does it receive as input at inference time?
6. Counting codec frames
The ENCODEC encoder emits embeddings at 75 Hz, one per frame. How many embedding frames does a -second clip of speech produce?
7. Counting the bits
A residual vector quantizer emits 8 codes per frame at 75 frames per second, and each code indexes a 1,024-entry codebook, costing 10 bits. How many bits per second of audio does the token stream carry?
8. Why two stages
VALL-E generates the first quantizer's codes with an autoregressive model and the remaining seven quantizers with non-autoregressive passes. What motivates the split?
9. Gradients through a lookup
Nearest-codeword assignment in vector quantization is not differentiable, which blocks end-to-end codec training. How does the straight-through estimator resolve this?