Say More with Less: Variable‑Frame‑Rate Speech Tokenization
via Adaptive Clustering and Implicit Duration Coding

Current speech tokenizers typically assign a fixed number of tokens per second, regardless of the varying information density or temporal fluctuations in the speech signal. This uniform token allocation mismatches the intrinsic structure of speech, where information is distributed unevenly over time. To address this, we propose VARSTok, a VAriable-frame-Rate Speech Tokenizer that adapts token allocation based on local feature similarity. VARSTok introduces a temporal-aware clustering algorithm that segments speech into variable-length units, followed by vector quantization. To support downstream language modeling without auxiliary duration predictors, we further design an implicit duration encoding scheme that integrates both content and temporal span into a single token index. Experimental results show that VARSTok achieves superior reconstruction naturalness and improved semantic representation compared to fixed-rate baselines while using fewer tokens. Additionally, it yields lower or comparable word error rates and better naturalness in downstream text-to-speech task. To the best of our knowledge, this is the first work to demonstrate that fully dynamic, variable-rate speech tokenizer can be directly and effectively integrated into downstream speech language models.