Trends · 研究趋势
数据来自 Hugging Face 论文的 AI 提取关键词,按月统计研究方向的增长与热度。
Shuming Ma, Hongyu Wang, Lingxiao Ma +7 authors
A 1-bit LLM variant, BitNet b1.58, achieves comparable performance to full-precision models with reduced computational costs and introduces new scaling laws and hardware design opportunities.
Tianzhu Ye, Li Dong, Yuqing Xia +4 authors
Diff Transformer improves large language models by selectively focusing attention on relevant context and reducing noise, leading to better performance in scaling, long-context modeling, key information retrieval, and in-context learning.
Yutao Sun, Li Dong, Shaohan Huang +5 authors
Retentive Network (RetNet) offers a foundation for large language models with parallel training, low-cost inference, and efficient long-sequence modeling through a retention mechanism supporting parallel, recurrent, and chunkwise recurrent paradigms.
Zhichao Wang, Dongyang Ma, Xinting Huang +6 authors
AutoDeco, a novel architecture, enables end-to-end generation by dynamically predicting decoding parameters, outperforming default strategies and achieving performance close to an oracle-tuned baseline.
Lianghua Huang, Zhifan Wu, Wei Wang +21 authors
Wan-Streamer is a unified, end-to-end multimodal model that enables real-time audio-visual interaction through causal attention mechanisms and integrated processing of visual, audio, and text modalities.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey +530 authors
Llama 3, a multilingual and multi-modal language model with 405B parameters, achieves competitive performance across tasks including image, video, and speech recognition when integrated through a compositional approach.
Opher Lieber, Barak Lenz, Hofit Bata +19 authors
Jamba, a hybrid Transformer-Mamba MoE model, achieves state-of-the-art performance with high throughput and minimal memory usage, supporting large context lengths.
Adam Polyak, Amit Zohar, Andrew Brown +85 authors
Movie Gen, a suite of foundation models, generates high-quality videos with synchronized audio, excelling in various tasks through architectural, training, and technical innovations.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo +22 authors
Emu3, a transformer-based multimodal model trained exclusively with next-token prediction, outperforms existing diffusion and compositional models in generation and perception tasks.
Sihan Xu, Ziqiao Ma, Wenhao Chai +5 authors
Generative pretraining using next embedding prediction outperforms traditional self-supervised methods in visual learning tasks, achieving high accuracy on ImageNet and effective transfer to semantic segmentation.
Kwanyoung Kim, Byeongsu Sim
PLADIS leverages sparse attention in cross-attention layers to enhance pre-trained text-to-image diffusion models, improving text alignment and human preference without additional training.
Jiayu Ding, Shuming Ma, Li Dong +4 authors
LongNet, a Transformer variant with dilated attention, enables scaling to over 1 billion tokens with linear computational complexity and maintains performance on shorter sequences.
Yaroslav Aksenov, Nikita Balagansky, Sofia Maria Lo Cicero Vaina +3 authors
A modification to the Based model kernel enhances its in-context learning capabilities, outperforming existing subquadratic architectures in language modeling tasks on the Pile dataset.
Vishesh Tripathi, Abhay Kumar
Grouped Query Experts (GQE) improves Transformer efficiency by selectively activating query heads based on token content while maintaining key-value cache benefits of grouped-query attention.
Chunting Zhou, Lili Yu, Arun Babu +7 authors
Transfusion combines language modeling and diffusion training to develop a multi-modal transformer model that scales effectively across text and image data, matching the performance of specialized models.
Daniel Goldstein, Fares Obeid, Eric Alcaide +2 authors
A hybrid Linear Attention/Transformer model, GoldFinch, efficiently generates a compressed KV-Cache for large context lengths with linear time and space complexity, outperforming previous models.
Roee Hendel, Mor Geva, Amir Globerson
In-Context Learning in Large Language Models can be understood as compressing a training set into a task vector that modulates a transformer for output generation.
Junxiong Wang, Daniele Paliotta, Avner May +2 authors
Large-scale Transformer models can be distilled into linear RNNs using attention layer weights, achieving competitive performance with enhanced deployment efficiency.
Lianghua Huang, Zhi-Fan Wu, Yupeng Shi +23 authors
Wan-Streamer v0.2 enhances audio-visual interaction by increasing visual resolution while maintaining low latency through optimized thinker-performer architecture with multi-GPU parallel processing.
Qihang Fan, Huaibo Huang, Mingrui Chen +2 authors
The proposed RMT model, combining RetNet and Transformer architectures, introduces explicit spatial distance priors and coordinate-wise decomposition to achieve exceptional performance in computer vision tasks.
北京市昌平区探索星信息技术及软件开发工作室
京ICP备2026059466号