CycleCodec: Pretrain-Free Factorized Neural Speech Codec via Cycle-Consistent Speaker Swapping

Factorized neural speech codecs often rely on supervision from pretrained self-supervised learning (SSL) encoders or automatic speech recognition (ASR) systems to disentangle content and speaker information. However, this dependency limits their applicability to low-resource languages where reliable teachers are unavailable during training. In this paper, we propose CycleCodec, a pretrain-free factorized codec trained entirely from scratch. We introduce a cycle-consistent speaker swapping strategy that enforces functional separability between time-varying content and global speaker streams via self-supervised reconstruction constraints. We further enhance the speaker representation using a query-based Transformer aggregator. Experiments on in-domain (English) and unseen languages (Mandarin and Vietnamese) show that CycleCodec significantly outperforms pretrain-free baselines and achieves more robust speaker-content disentanglement on unseen languages.