Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization

Zeyuan Liu; Jeonghye Kim; Xufang Luo; Dongsheng Li; Yuqing Yang

返回文献探索

Paper · arXiv 2602.23008

Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization

Zeyuan Liu, Jeonghye Kim, Xufang Luo, Dongsheng Li, Yuqing Yang

37 upvotesFebruary 26, 2026arXiv 预印本

AI 摘要

EMPO² is a hybrid reinforcement learning framework that enhances exploration for large language model agents by integrating memory mechanisms with on- and off-policy updates, demonstrating improved performance and adaptability in complex environments.

reinforcement learninglarge language model agentsexplorationmemory augmentationon-policy updatesoff-policy updatesScienceWorldWebShop

Abstract

Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO^2), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO^2 achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO^2 demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO^2 as a promising framework for building more exploratory and generalizable LLM-based agents.