Attention or Convolution: Transformer Encoders in Audio Language Models
  for Inference Efficiency

Sungho Jeon; Ching-Feng Yeh; Hakan Inan; Wei-Ning Hsu; Rashi Rungta; Yashar Mehdad; Daniel Bikel

返回文献探索

Paper · arXiv 2311.02772

Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiency

Sungho Jeon, Ching-Feng Yeh, Hakan Inan, Wei-Ning Hsu, Rashi Rungta, Yashar Mehdad, Daniel Bikel

5 upvotesNovember 5, 2023arXiv 预印本

AI 摘要

A simpler self-attention-only approach in pre-trained audio models matches the efficiency of more complex speech transformer models, especially with low-bit weight quantization.

self-supervisedpre-trainedaudio modelspeech transformerconvolutional modulesself-attention modulesASRefficiencyencoderlow-bit weight quantizationerror propagationquantized modules

Abstract

In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transformer encoders. These speech transformers rely on mixing convolutional modules with self-attention modules. They achieve state-of-the-art performance on ASR with top efficiency. We first show that employing these speech transformers as an encoder significantly improves the efficiency of pre-trained audio models as well. However, our study shows that we can achieve comparable efficiency with advanced self-attention solely. We demonstrate that this simpler approach is particularly beneficial with a low-bit weight quantization technique of a neural network to improve efficiency. We hypothesize that it prevents propagating the errors between different quantized modules compared to recent speech transformers mixing quantized convolution and the quantized self-attention modules.