EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for
  Speech-to-Speech LLMs

Yuhao Zhang; Yuhao Du; Zhanchen Dai; Xiangnan Ma; Kaiqi Kou; Benyou Wang; Haizhou Li

返回文献探索

Paper · arXiv 2509.09174

EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs

Yuhao Zhang, Yuhao Du, Zhanchen Dai, Xiangnan Ma, Kaiqi Kou, Benyou Wang, Haizhou Li

62 upvotesSeptember 11, 2025arXiv 预印本

AI 摘要

EchoX, a speech-to-speech large language model, addresses the acoustic-semantic gap by integrating semantic representations, preserving reasoning abilities, and achieving advanced performance on knowledge-based benchmarks.

speech-to-speech large language modelsSLLMstext-based large language modelsLLMsacoustic-semantic gapsemantic representationsspeech training targetsacoustic learningsemantic learningknowledge-based question-answering benchmarks

Abstract

Speech-to-speech large language models (SLLMs) are attracting increasing attention. Derived from text-based large language models (LLMs), SLLMs often exhibit degradation in knowledge and reasoning capabilities. We hypothesize that this limitation arises because current training paradigms for SLLMs fail to bridge the acoustic-semantic gap in the feature representation space. To address this issue, we propose EchoX, which leverages semantic representations and dynamically generates speech training targets. This approach integrates both acoustic and semantic learning, enabling EchoX to preserve strong reasoning abilities as a speech LLM. Experimental results demonstrate that EchoX, with about six thousand hours of training data, achieves advanced performance on multiple knowledge-based question-answering benchmarks. The project is available at https://github.com/FreedomIntelligence/EchoX.