Physics in Next-token Prediction

Hongjun An; Yiliang Song; Xuelong Li

返回文献探索

Paper · arXiv 2411.00660

Physics in Next-token Prediction

Hongjun An, Yiliang Song, Xuelong Li

14 upvotesNovember 1, 2024arXiv 预印本

AI 摘要

The study identifies laws of information capacity in Next-token Prediction and applies Landauer's Principle to relate auto-regressive model training to energy consumption.

Next-token PredictionFirst Law of Information CapacitySecond Law of Information Capacityauto-regressive modelsinformation transferLandauer's Principleinformation capacity

Abstract

We discovered the underlying physics in Next-token Prediction (NTP). We identified the law of information conservation within NTP and proposed the First Law of Information Capacity (IC-1), demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer. We also introduced Landauer's Principle into NTP, formulating the Second Law of Information Capacity (IC-2), which establishes the relationship between auto-regressive model training and energy consumption. Additionally, we presented several corollaries, which hold practical significance for production practices. Finally, we validated the compatibility and complementarity of our findings with existing theories.