S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

Shuo Shen; Guangran Cheng; Chengqi Lyu; Xiangsen Wang; Yanan Sun; Wenwei Zhang

返回文献探索

Paper · arXiv 2610.35518

S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

Shuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun, Wenwei Zhang

26 upvotesOctober 26, 2026arXiv 预印本

AI 摘要

S3AC introduces a stabilized actor-critic training method with difficulty-aware prompt sampling to improve long reasoning in large language models without relying on grouped rollouts.

reinforcement learning with verifiable rewardschain-of-thought reasoningactor-critictoken-level credit assignmentvalue headDifficulty-Aware Prompt Samplingmixture-of-expertsGRPO

Abstract

Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.