Associative Recurrent Memory Transformer

Ivan Rodkin; Yuri Kuratov; Aydar Bulatov; Mikhail Burtsev

返回文献探索

Paper · arXiv 2407.04841

Associative Recurrent Memory Transformer

Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, Mikhail Burtsev

35 upvotesJuly 5, 2024arXiv 预印本

AI 摘要

The Associative Recurrent Memory Transformer effectively handles long sequences by combining transformer self-attention with segment-level recurrence, achieving high accuracy in long-context tasks like BABILong.

transformer self-attentionsegment-level recurrenceAssociative Recurrent Memory TransformerBABILong benchmark

Abstract

This paper addresses the challenge of creating a neural architecture for very long sequences that requires constant time for processing new information at each time step. Our approach, Associative Recurrent Memory Transformer (ARMT), is based on transformer self-attention for local context and segment-level recurrence for storage of task specific information distributed over a long context. We demonstrate that ARMT outperfors existing alternatives in associative retrieval tasks and sets a new performance record in the recent BABILong multi-task long-context benchmark by answering single-fact questions over 50 million tokens with an accuracy of 79.9%. The source code for training and evaluation is available on github.