LEPO: Latent Reasoning Policy Optimization for Large Language Models

摘要

Recently, latent reasoning has been introduced into large language models (LLMs) to leverage rich information within a continuous space.However, without stochastic sampling, these methods inevitably collapse to deterministic inference, failing to discover diverse reasoning paths.To bridge the gap, we inject controllable stochasticity into latent reasoning via Gumbel-Softmax, restoring LLMs’ exploratory capacity and enhancing their compatibility with Reinforcement Learning (RL).Building on this, we propose **L**atent R**e**asoning **P**olicy **O**ptimization (**LEPO**), a novel framework that applies RL directly to continuous latent representations.Specifically, in rollout stage, LEPO maintains stochasticity to enable diverse trajectory sampling, while in optimization stage, LEPO constructs a unified gradient estimation for both latent representations and discrete tokens.