Revisiting Learning Paradigms for Multimedia Data Generation

摘要

With the development of deep learning, multimedia data generation (e.g., image generation, audio synthesis, music composition, and video generation) has attracted a lot of attention. Deep learning methods for data generation usually build a mapping from source condition X to target data Y. The target Y (e.g., image, speech, music, video) is usually high-dimensional and complex, and contains rich information not exist in source data, which hinders the effective and efficient learning on the source-target mapping. Representation learning has achieved rapid progress in the past decade, which is beneficial for data understanding tasks. However, traditional representation learning cannot address the challenges faced by multimedia data generation tasks. This tutorial revisits the learning paradigms for data generation and introduces a paradigm called regeneration learning that can improve the effectiveness and efficiency of multimedia data generation. We show that a variety of tasks in multimedia data generation (e.g., image generation, speech synthesis, music composition, video generation) can benefit from this regeneration learning paradigm, and a lot of recent popular data generation models (e.g., DALL-E 1/2, Stable Diffusion, AudioLM, NaturalSpeech 2, MusicLM) can be covered by this learning paradigm.