Investigating Decoder-only Large Language Models for Speech-to-text
  Translation

Chao-Wei Huang; Hui Lu; Hongyu Gong; Hirofumi Inaguma; Ilia Kulikov; Ruslan Mavlyutov; Sravya Popuri

返回文献探索

Paper · arXiv 2407.03169

Investigating Decoder-only Large Language Models for Speech-to-text Translation

Chao-Wei Huang, Hui Lu, Hongyu Gong, Hirofumi Inaguma, Ilia Kulikov, Ruslan Mavlyutov, Sravya Popuri

9 upvotesJuly 3, 2024arXiv 预印本

AI 摘要

Decoder-only large language models integrated with speech-to-text translation achieve state-of-the-art performance without proprietary data through parameter-efficient fine-tuning.

decoder-only LLMsspeech-to-text translationCoVoST 2FLEURSparameter-efficient fine-tuning

Abstract

Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT.