Star Attention: Efficient LLM Inference over Long Sequences

Shantanu Acharya; Fei Jia; Boris Ginsburg

返回文献探索

Paper · arXiv 2411.17116

Star Attention: Efficient LLM Inference over Long Sequences

Shantanu Acharya, Fei Jia, Boris Ginsburg

53 upvotesNovember 26, 2024arXiv 预印本

AI 摘要

Star Attention improves inference efficiency of large language models on long sequences by using block-sparse approximation, reducing memory and time without significant accuracy loss.

Transformer-based Large Language Models (LLMs)self-attention mechanismblock-sparse approximationStar Attentionblockwise-local attentionsequence-global attentionmemory requirementsinference timeaccuracy

Abstract

Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, a two-phase block-sparse approximation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise-local attention across hosts, in parallel. In the second phase, query and response tokens attend to all prior cached tokens through sequence-global attention. Star Attention integrates seamlessly with most Transformer-based LLMs trained with global attention, reducing memory requirements and inference time by up to 11x while preserving 95-100% of accuracy.