DiffuFuse: Diffusion-Driven Dual-Stream Fusion Framework for Multimodal Sentiment Analysis

摘要

Multimodal Sentiment Analysis (MSA) aims to integrate textual, audio, and visual data to capture nuanced sentimental cues. Although text dominates in existing approaches, audio and visual modalities inherently contain both shared semantics (overlapping with text) and private semantics. Existing methods struggle to precisely find semantic boundaries and lack explicit mechanisms for modeling interaction between shared/private semantics and different modalities. To address this, we propose DiffuFuse, a framework that uses a diffusion denoising model to leverage textual information to predict shared semantic features, dynamically and adaptively delineate semantic boundaries for non-textual features, and employs a dual-stream fusion strategy to accurately model the interactions between different modalities and semantic types. Finally, adopt an orthogonal projection method to reduce redundancy and eliminate overlapping information between the two streams. DiffuFuse is evaluated on the MOSI and MOSEI datasets, and the experimental results demonstrate that our proposed DiffuFuse achieves superior performance.