Retentive Network: A Successor to Transformer for Large Language Models
Yutao Sun, Li Dong, Shaohan Huang +5 authors
Retentive Network (RetNet) offers a foundation for large language models with parallel training, low-cost inference, and efficient long-sequence modeling through a retention mechanism supporting parallel, recurrent, and chunkwise recurrent paradigms.