Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

摘要

Reinforcement learning (RL) is widely used to improve large language models (LLMs) on reasoning tasks, and asynchronous RL training is attractive because it increases end-to-end throughput. However, for widely adopted critic-free policy-gradient methods such as REINFORCE and GRPO, high asynchrony makes the policy-gradient estimator **markedly higher variance**: stale off-policy rollouts induce heavy-tailed importance ratios, causing a small fraction of samples to dominate each update. This amplification makes gradients noisy and learning unstable relative to matched on-policy training. Across math and reasoning benchmarks, we find collapse is preceded by sharp drops in effective sample size (ESS) and unstable gradient norms. Motivated by this diagnosis, we propose **V**ariance **C**ontrolled **P**olicy **O**ptimization (VCPO), a drop-in stabilization method for REINFORCE/GRPO-style algorithms that (i) **rescales learning rate according to effective sample size** to dampen unreliable updates, and (ii) applies a **closed-form minimum-variance baseline** for the off-policy setting, avoiding an auxiliary value model and adding minimal overhead. Empirically, VCPO substantially improves robustness in highly asynchronous regimes across models sizes and tasks, reducing long-context, multi-turn training compute by 1.96×. Overall, our results demonstrate explicitly controlling policy-gradient variance is key to making asynchronous RL reliable at scale.