Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic
  Confidence

Amirhosein Ghasemabadi; Keith G. Mills; Baochun Li; Di Niu

返回文献探索

Paper · arXiv 2505.20325

Guided by Gut: Efficient Test-Time Scaling with Reinforced Intrinsic Confidence

Amirhosein Ghasemabadi, Keith G. Mills, Baochun Li, Di Niu

46 upvotesMay 23, 2025arXiv 预印本

AI 摘要

The Guided by Gut (GG) framework enhances LLM reasoning efficiently using intrinsic signals and token-level confidence, outperforming PRM-based methods with faster inference and lower memory usage.

Test-Time Scaling (TTS)Large Language Model (LLM)Process Reward Models (PRMs)Best-of-N (BoN)tree searchintrinsic LLM signalstoken-level confidencestep noveltytargeted reinforcement learning fine-tuning

Abstract

Test-Time Scaling (TTS) methods for enhancing Large Language Model (LLM) reasoning often incur substantial computational costs, primarily due to extensive reliance on external Process Reward Models (PRMs) or sampling methods like Best-of-N (BoN). This paper introduces Guided by Gut (GG), an efficient self-guided TTS framework that achieves PRM-level performance without costly external verifier models. Our method employs a lightweight tree search guided solely by intrinsic LLM signals, token-level confidence and step novelty. One critical innovation is improving the reliability of internal confidence estimates via a targeted reinforcement learning fine-tuning phase. Empirical evaluations on challenging mathematical reasoning benchmarks demonstrate that GG enables smaller models (e.g., 1.5B parameters) to achieve accuracy matching or surpassing significantly larger models (e.g., 32B-70B parameters), while reducing GPU memory usage by up to 10x. Compared to PRM-based methods, GG achieves comparable accuracy with 8x faster inference speeds and 4-5x lower memory usage. Additionally, GG reduces KV cache memory usage by approximately 50% compared to the BoN strategy, facilitating more efficient and practical deployment of TTS techniques.