Revisiting Multimodal Positional Encoding in Vision-Language Models

Jie Huang; Xuejing Liu; Sibo Song; Ruibing Hou; Hong Chang; Junyang Lin; Shuai Bai

返回文献探索

Paper · arXiv 2510.23095

Revisiting Multimodal Positional Encoding in Vision-Language Models

Jie Huang, Xuejing Liu, Sibo Song, Ruibing Hou, Hong Chang, Junyang Lin, Shuai Bai

23 upvotesOctober 27, 2025arXiv 预印本

AI 摘要

A comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) leads to the proposal of Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), which improve multimodal understanding in vision-language models.

multimodal position encodingRotary Positional Embedding (RoPE)position designfrequency allocationpositional coherencefull frequency utilizationpreservation of textual priorsMulti-Head RoPE (MHRoPE)MRoPE-Interleave (MRoPE-I)multimodal understanding

Abstract

Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors-ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding. Code will be avaliable at https://github.com/JJJYmmm/Multimodal-RoPEs.