Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and
  Dialogue Abilities

Zhifeng Kong; Arushi Goel; Rohan Badlani; Wei Ping; Rafael Valle; Bryan Catanzaro

返回文献探索

Paper · arXiv 2402.01831

Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities

Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, Bryan Catanzaro

17 upvotesFebruary 2, 2024arXiv 预印本

AI 摘要

Audio Flamingo, a novel audio language model, achieves strong audio understanding, in-context learning, and multi-turn dialogue capabilities through advanced training techniques and architecture design.

audio language modelaudio understandingin-context learningmulti-turn dialoguetraining techniquesarchitecture design

Abstract

Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs. In this paper, we propose Audio Flamingo, a novel audio language model with 1) strong audio understanding abilities, 2) the ability to quickly adapt to unseen tasks via in-context learning and retrieval, and 3) strong multi-turn dialogue abilities. We introduce a series of training techniques, architecture design, and data strategies to enhance our model with these abilities. Extensive evaluations across various audio understanding tasks confirm the efficacy of our method, setting new state-of-the-art benchmarks.