Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Chinese Academy of Sciences · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · miHoYo · University of the Chinese Academy of Sciences · Institute of automation, Chinese academy of science, Chinese Academy of Sciences · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Chinese Academy of Sciences · National University of Singapore
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have empowered large language models (LLMs) to tackle challenging reasoning tasks such as mathematics and programming. Despite its promise, the RLVR paradigm poses significant challenges, as existing methods often suffer from sparse reward signals and unstable policy gradient updates, inherent to RL-based approaches. To address the challenges, we propose **PACS**, a novel RLVR framework that achieves im**P**licit **A**ctor **C**ritic coupling via a **S**upervised learning framework. By treating the outcome reward as a predictable label, we reformulate the RLVR problem into a supervised learning task over a score function parameterized by the policy model and optimized using cross-entropy loss. A detailed gradient analysis shows that this supervised formulation inherently recovers the classical policy gradient update while providing more stable and efficient training. Extensive experiments demonstrate that PACS significantly outperforms strong open-source models and RLVR baselines, yielding substantial average gains of **+8.26%** (4B) and **+9.57%** (8B) over base models offering a promising avenue for LLMs post-training with verifiable rewards.