First Return, Entropy-Eliciting Explore

Tianyu Zheng; Tianshun Xing; Qingshui Gu; Taoran Liang; Xingwei Qu; Xin Zhou; Yizhi Li; Zhoufutu Wen; Chenghua Lin; Wenhao Huang; Qian Liu; Ge Zhang; Zejun Ma

Paper · arXiv 2507.07017

First Return, Entropy-Eliciting Explore

Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, Zejun Ma

24 upvotesJuly 9, 2025arXiv 预印本

AI 摘要

FR3E, a structured exploration framework, enhances LLM reasoning by providing targeted guidance at high-uncertainty decision points, leading to more stable training and accurate responses.

Reinforcement Learning from Verifiable RewardsLarge Language ModelsFR3Estructured explorationhigh-uncertainty decision pointstargeted rolloutssemantically grounded intermediate feedbackmathematical reasoning benchmarksAIME24

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration.