MusFlow: Multimodal Music Generation via Conditional Flow Matching

摘要

Music generation aims to create music segments that align with human aesthetics based on diverse conditions. Despite advancements in generating music from specific textual descriptions (e.g., style, genre, instruments), the practical application is still hindered by ordinary users' limited expertise to write accurate prompts. To bridge this application gap, this paper introduces MusFlow, a novel multimodal music generation model using Conditional Flow Matching (CFM). We employ multiple Multi-Layer Perceptrons to align multimodal conditions into the audio's CLAP embedding space. CFM is trained to reconstruct the compressed Mel-spectrogram in the VAE latent space guided by aligned feature embedding. MusFlow can generate music from images, story texts, and music captions. To collect data for model training, inspired by multi-agent collaboration, we construct an intelligent annotation workflow centered around a fine-tuned Qwen2-VL model. Using this workflow, we build a new multimodal music dataset, MMusSet, with each sample containing a quadruple of image, story text, music caption, and music piece. We conduct four sets of experiments: image-to-music, story-to-music, caption-to-music, and multimodal music generation. Experimental results demonstrate that MusFlow can generate high-quality music pieces from multimoal conditions. We hope this work can advance the application of music generation in multimedia field, making music creation more accessible. Our generated samples are available at https://anonymous22356.github.io/musflow.github.io/