A Theoretical Proof of Dynamic Multimodal Fusion Exacerbates Modality Greedy

摘要

In recent years, numerous studies have proposed uncertainty-guided multimodal learning to adapt to dynamic relationships between different modalities. Dynamic multimodal fusion enables more dominant modalities to receive greater weight during the fusion process, thereby preventing the influence of spurious features from less reliable modalities on decision-making. However, there is No free lunch. We observe that the introduction of dynamic fusion during training exacerbates the model's tendency toward Greedy (a phenomenon known to induce decision shortcuts in multimodal learning). This results in a model that does not fully take advantage of the lower quality modalities. In this paper, we provide a theoretical analysis showing that dynamic fusion intensifies Greedy, and we present experimental results that support this observation. In summary, this paper explains the Greedy risk in dynamic multimodal learning from both theoretical and experimental perspectives, serving as a cautionary reminder for researchers when employing dynamic multimodal learning. Our code is available at https://github.com/d-xr/GreedyDynFusion.