Trends · 研究趋势
数据来自 Hugging Face 论文的 AI 提取关键词,按月统计研究方向的增长与热度。
Guan Wang, Changling Liu, Chenyu Wang +6 authors
A Hierarchical Recurrent Model architecture with specialized training on instruction-response pairs achieves competitive language modeling performance with significantly reduced computational requirements compared to traditional Transformer-based approaches.
Aeree Cho, Grace C. Kim, Alexander Karpekov +5 authors
Transformer Explainer is an interactive visualization tool that allows non-experts to understand the inner workings of the GPT-2 model through real-time experimentation and visualization in a web browser.
Jiachen Zhu, Xinlei Chen, Kaiming He +2 authors
Dynamic Tanh (DyT) replaces normalization layers in Transformers, achieving equivalent or superior performance without hyperparameter tuning across various tasks.
Bo Peng, Ruichong Zhang, Daniel Goldstein +12 authors
RWKV-7 "Goose" achieves state-of-the-art performance in multilingual tasks with optimal memory and inference efficiency, exceeding Transformer capabilities in complexity.
Roman Abramov, Felix Steinbauer, Gjergji Kasneci
Grokking with synthetic data augmentation in knowledge graphs enhances Transformer models' multi-step factual reasoning on real-world tasks, achieving high accuracy on benchmarks.
Timothée Darcet, Maxime Oquab, Julien Mairal +1 authors
Additional input tokens in Vision Transformers mitigate artifacts in feature maps, enhancing performance and enabling smoother visual processing.
Zunhai Su, Hengyuan Zhang, Wei Wu +17 authors
Transformers face challenges from Attention Sink phenomenon where excessive attention focuses on uninformative tokens, impacting interpretability and performance, necessitating comprehensive research survey addressing fundamental usage, mechanistic understanding, and strategic mitigation approaches.
Maciej Pióro, Kamil Ciebiera, Krystian Król +2 authors
A combination of Mixture of Experts (MoE) with State Space Models (SSMs) improves model efficiency and performance compared to both standalone SSMs and MoE-enhanced Transformers.
Tri Dao, Albert Gu
A new framework connects state-space models and transformers, leading to a faster architecture for language modeling.
Soham De, Samuel L. Smith, Anushan Fernando +14 authors
Hawk and Griffin, models incorporating gated linear recurrences, achieve high performance with lower computational resources and better hardware efficiency compared to existing architectures.
Sean McLeish, Arpit Bansal, Alex Stein +8 authors
Transformers achieve state-of-the-art performance on large arithmetic tasks and other reasoning tasks by addressing positional tracking with embeddings and integrating architectural modifications.
Duy-Kien Nguyen, Mahmoud Assran, Unnat Jain +3 authors
Vanilla Transformers achieve high performance in computer vision tasks by treating individual pixels as tokens, challenging the conventional use of local neighborhoods.
Lucas Lehnert, Sainbayar Sukhbaatar, Paul Mcvay +2 authors
Searchformer, a Transformer model, outperforms traditional $A^*$ search by optimizing Sokoban puzzle solutions with fewer steps and demonstrates superior performance on maze navigation and more complex tasks.
Wilfried Bounsi, Borja Ibarz, Andrew Dudzik +5 authors
A novel TransNAR model combines Transformer-based language understanding with graph neural network solvers to enhance algorithmic reasoning.
Dongseong Hwang, Weiran Wang, Zhuoyuan Huo +2 authors
Feedback Attention Memory (FAM) enhances Transformer architecture by enabling long-context processing without additional weights, significantly improving performance on large sequences across various model sizes.
Boshi Wang, Xiang Yue, Yu Su +1 authors
Transformers can learn implicit reasoning through extensive training, with varying degrees of generalization for different reasoning types, and parametric memory proves more effective for complex reasoning tasks compared to non-parametric memory.
Hao Liu, Wilson Yan, Matei Zaharia +1 authors
Large-scale training of a transformer model for long video and language sequences using RingAttention and masked sequence packing improves performance on retrieval and long-form understanding tasks.
Liliang Ren, Yang Liu, Yadong Lu +3 authors
Samba, a hybrid architecture combining State Space Models and Sliding Window Attention, efficiently handles long-range context with lower computational complexity and better memory recall than pure attention or SSM models.
Matanel Oren, Michael Hassid, Yossi Adi +1 authors
Decoder-only transformers can be conceptualized as finite multi-state RNNs, and a new cache compression technique, TOVA, significantly reduces their computational cost while maintaining high performance.
Jiacheng Liu, Sewon Min, Luke Zettlemoyer +2 authors
Modernized n-gram models, specifically $\infty$-gram models, enhance text analysis and neural LLMs by providing high accuracy in next-token prediction and reducing language modeling perplexities.
北京市昌平区探索星信息技术及软件开发工作室
京ICP备2026059466号