MIDE: Multimodal Dialogue Emotion Recognition via Mutual Information Enhancement and Dynamic Modality Selection

摘要

Multimodal dialogue emotion recognition seeks to integrate information from text, audio, and video to accurately determine the emotional state of each utterance. However, prevailing approaches often depend on fixed fusion strategies, failing to account for quality variations across modalities. Consequently, noise from less informative modalities can degrade overall performance. Moreover, most models treat utterances as independent units for static classification, overlooking the dynamic evolution of emotions throughout a dialogue. To overcome these limitations, we propose MIDE, a novel approach that leverages mutual information enhancement and dynamic modality slection for multimodal emotion recognition. Our approach dynamically selects high-quality modality pairs for fusion, minimizing interference from noisy or redundant sources. Specifically, it employs mutual information maximization to achieve cross-modal semantic alignment and incorporates an autonomous modality selection mechanism to assess inter-modal compatibility. Furthermore, an emotion transition prediction module, implemented with a Gated Recurrent Unit (GRU), captures temporal emotional dependencies, enabling joint optimization of static emotion classification and dynamic emotion trajectory prediction. Extensive experiments on the IEMOCAP and MELD datasets demonstrate that MIDE significantly surpasses existing models in accuracy and robustness, highlighting its strength in adaptive fusion for complex dialogue scenarios.