PolyVoice: Language Models for Speech to Speech Translation
ByteDance · University of Science and Technology of China · Harbin Engineering University · ByteDance AI Lab · Institute of automation, Chinese academy of science, Chinese Academy of Sciences · Delft University of Technology · SUN YAT-SEN UNIVERSITY · Beijing University of Aeronautics and Astronautics · Peking University · Sudan University for Science and Technology · Bytedance Inc. · Bytedance · Louisiana State University · iscas · , Chinese Academy of Sciences · ByteDance, Inc.
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
With the huge success of GPT models in natural language processing, there is a growing interest in applying language modeling approaches to speech tasks.Currently, the dominant architecture in speech-to-speech translation (S2ST) remains the encoder-decoder paradigm, creating a need to investigate the impact of language modeling approaches in this area. In this study, we introduce PolyVoice, a language model-based framework designed for S2ST systems. Our framework comprises three decoder-only language models: a translation language model, a duration language model, and a speech synthesis language model. These language models employ different types of prompts to extract learned information effectively. By utilizing unsupervised semantic units, our framework can transfer semantic information across these models, making it applicable even to unwritten languages. We evaluate our system on Chinese $\rightarrow$ English and English $\rightarrow$ Spanish language pairs. Experimental results demonstrate that \method outperforms the state-of-the-art encoder-decoder model, producing voice-cloned speech with high translation and audio quality.Speech samples are available at https://polyvoice.github.io.