ERVQ: Enhanced Residual Vector Quantization with Intra-and-Inter-Codebook Optimization for Neural Audio Codecs

Current neural audio codecs typically use residual vector quantization (RVQ) to discretize speech signals. However, they often experience codebook collapse, which reduces the effective codebook size and leads to suboptimal per formance. To address this problem, we introduce ERVQ, Enhanced Residual Vector Quantization, a novel enhance ment strategy for the RVQ framework in neural audio codecs. ERVQ mitigates codebook collapse and boosts codec perfor mance through both intra- and inter-codebook optimization. Intra-codebook optimization incorporates an online cluster ing strategy and a code balancing loss to ensure balanced and efficient codebook utilization. Inter-codebook optimiza tion improves the diversity of quantized features by mini mizing the similarity between successive quantizations. Our experiments show that ERVQ significantly enhances audio codec performance across different models, sampling rates, and bitrates, achieving superior quality and generalization capabilities. It also achieves 100% codebook utilization on one of the most advanced neural audio codecs. Further experiments indicate that audio codecs improved by the ERVQ strategy can improve unified speech-and-text large language models (LLMs). Specifically, there is a notable improvement in the naturalness of generated speech in downstream zero shot text-to-speech tasks.