Beyond Homogeneous Attention: Memory-Efficient LLMs via
  Fourier-Approximated KV Cache

Xiaoran Liu; Siyang He; Qiqi Wang; Ruixiao Li; Yuerong Song; Zhigeng Liu; Linlin Li; Qun Liu; Zengfeng Huang; Qipeng Guo; Ziwei He; Xipeng Qiu

Paper · arXiv 2506.11886

Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache

Xiaoran Liu, Siyang He, Qiqi Wang, Ruixiao Li, Yuerong Song, Zhigeng Liu, Linlin Li, Qun Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, Xipeng Qiu

20 upvotesJune 13, 2025arXiv 预印本

AI 摘要

FourierAttention is a training-free framework that enhances memory efficiency in Large Language Models by compressing long-context-insensitive transformer head dimensions using orthogonal Fourier bases, while maintaining accuracy.

FourierAttentiontransformer head dimensionsLongBenchNeedle-In-A-HaystackFlashFourierAttentionorthogonal Fourier baseslong-context accuracy

Abstract

Large Language Models struggle with memory demands from the growing Key-Value (KV) cache as context lengths increase. Existing compression methods homogenize head dimensions or rely on attention-guided token pruning, often sacrificing accuracy or introducing computational overhead. We propose FourierAttention, a training-free framework that exploits the heterogeneous roles of transformer head dimensions: lower dimensions prioritize local context, while upper ones capture long-range dependencies. By projecting the long-context-insensitive dimensions onto orthogonal Fourier bases, FourierAttention approximates their temporal evolution with fixed-length spectral coefficients. Evaluations on LLaMA models show that FourierAttention achieves the best long-context accuracy on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel, FlashFourierAttention, is designed to optimize memory via streamlined read-write operations, enabling efficient deployment without performance compromise.