Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding
MBZUAI · University of California, San Diego · The University of Tokyo · Mohamed bin Zayed University of Artificial Intelligence · Computer Science Department, Stanford University · The Chinese University of Hong Kong, Shenzhen · University of California, San Diego, University of California, San Diego · School of Computer Science, Carnegie Mellon University · Petuum Inc. and CMU
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
The vast applications of deep generative models are anchored in three core capabilities---*generating* new instances, *reconstructing* inputs, and learning compact *representations*---across various data types, such as discrete text/protein sequences and continuous images. Existing model families, like variational autoencoders (VAEs), generative adversarial networks (GANs), autoregressive models, and (latent) diffusion models, generally excel in specific capabilities and data types but fall short in others. We introduce *Generalized* ***E****ncoding*-***D****ecoding ****D****iffusion ****P****robabilistic ****M****odels* (EDDPMs) which integrate the core capabilities for broad applicability and enhanced performance. EDDPMs generalize the Gaussian noising-denoising in standard diffusion by introducing parameterized encoding-decoding. Crucially, EDDPMs are compatible with the well-established diffusion model objective and training recipes, allowing effective learning of the encoder-decoder parameters *jointly* with diffusion. By choosing appropriate encoder/decoder (e.g., large language models), EDDPMs naturally apply to different data types. Extensive experiments on text, proteins, and images demonstrate the flexibility to handle diverse data and tasks and the strong improvement over various existing models. Code is available at https://github.com/guangyliu/EDDPM .