One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

摘要

Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous implementations leave GPUs idle during pipeline bubbles, wasting computational resources. Asynchronous Pipeline Parallelism approaches effectively eliminate these bubbles, maximizing throughput at the cost of gradient staleness. Unlike other schemes, PipeDream-2BW ensures a constant one-step gradient delay regardless of pipeline depth. However, its widespread adoption remains limited due to the common belief that optimizing under staleness is fundamentally unstable. In this work, we challenge this assumption, demonstrating that convergence degradation is largely an artifact of optimizer choice rather than an intrinsic limitation. We provide the first comprehensive analysis showing that while AdamW, the predominant optimizer at the time when PipeDream-2BW was introduced, indeed suffers from severe degradation, recent methods like Muon exhibit inherent robustness under a one step delay. We support this with theoretical analysis and introduce an optimizer-agnostic Error-Feedback mechanism to further mitigate delay effects. Extensive evaluation on models up to 10B parameters confirms that our strategies bridge the performance gap with synchronous training, enabling the practical deployment of asynchronous pipeline parallelism at scale.