RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
University of Washington · Department of Computer Science, University of Washington · University of Illinois at Urbana-Champaign · Princeton University · CSE, University of Washington · University of Washington, Seattle · UCLA Computer Science Department, University of California, Los Angeles · University of Washington, Google DeepMind · Department of Computer Science, University of Illinois Urbana-Champaign
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model's capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym, we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM's original RL training yields only a 0.49% average absolute gain despite using over 3x more compute. We will release our code publicly.