Can Simple Denoising Improve Uniform State Diffusion Models?
Pennsylvania State University · Zhejiang University · State University of New York at Buffalo · HKU · University of Washington · City University of Hong Kong · Penn State / Bytedance Seed · The Pennsylvania State University · Penn State
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Recent Uniform-state Diffusion Models (USDMs), initialized from a uniform prior, offer the promise of fast text generation due to their inherent self-correction ability compared to masked diffusion models. However, they still rely on complex loss formulations with additional computational overhead, which hinders scalability. In this work, we explore a simplified denoising-based loss for USDMs that optimizes only noise-replaced tokens, stabilizing training while matching the performance of prior methods with more complex objectives. In addition, we introduce an efficient regularization term to mitigate corruption toward uniform output distributions, which further improves performance. We demonstrate the effectiveness and efficiency of our simple and improved loss formulations by pretraining models on widely used text datasets for USDMs. More importantly, our conclusions scale to larger models, showing strong potential for large-scale training.