FlowRL: Matching Reward Distributions for LLM Reasoning
Shanghai Jiaotong University · Microsoft Research · Peking University · Tsinghua University · Tsinghua University, Tsinghua University · Computer Science, Stanford University · Fudan University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory Tsinghua University · Johns Hopkins University
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
We propose FlowRL: matching the full reward distribution via flow balancing instead of solely maximizing rewards in large language model (LLM) reinforcement learning (RL). Recent advanced reasoning models adopt reward-maximizing methods (e.g., PPO and GRPO), which tend to over-optimize dominant reward signals while neglecting less frequent but valid reasoning paths, thus reducing diversity. In contrast, we transform scalar rewards into a normalized target distribution using a learnable partition function, and then minimize the reverse KL divergence between the policy and the target distribution. We implement this idea as a flow-balanced optimization method that promotes diverse exploration and generalizable reasoning trajectories. We conduct experiments on both math and code reasoning tasks: FlowRL achieves a significant average improvement of $10.0\%$ over GRPO and $5.1\%$ over PPO on math benchmarks, and performs consistently better on code reasoning tasks. These results highlight reward distribution-matching as a key step toward efficient exploration and diverse reasoning in LLM reinforcement learning.