In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs
  Miss

Yuri Kuratov; Aydar Bulatov; Petr Anokhin; Dmitry Sorokin; Artyom Sorokin; Mikhail Burtsev

返回文献探索

Paper · arXiv 2402.10790

In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs Miss

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, Mikhail Burtsev

42 upvotesFebruary 16, 2024arXiv 预印本

AI 摘要

Recursively augmented GPT-2 fine-tuning significantly extends the processing capability of generative models to handle long document sequences beyond $10^6$ elements.

generative transformer modelsBABILong benchmarkGPT-4RAGfine-tuningrecurrent memoryopen neural network models

Abstract

This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to 10^4 elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to 10^7 elements. This achievement marks a substantial leap, as it is by far the longest input processed by any open neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences.