Explaining Listener Reactions: Personality-Guided Facial Response Generation with Cross-Modal Attention

摘要

Generating diverse and contextually appropriate facial reactions remains a significant challenge due to variability in individual responses, limited explainability, and insufficient modeling of contextual cues. In this study, we propose a multimodal framework that integrates behavioral memory, dynamic attention control, and cognitive style modeling to generate personalized and psychologically grounded facial reactions in dyadic interactions. Our method models the causal link between speaker behavior and listener response by incorporating frame-level behavioral cues, personality traits, and cognitive processing styles. The proposed system consists of three core components: a behavioral memory module that captures temporal context across conversation turns; a Personalized Personality Recognition Style (PPRS) module that infers cognitive tendencies via dual-path learning based on the Big Five personality traits; and a transformer-based generative module equipped with diffusion modeling and context-aware attention gating. This design enables the generation of expressive, individualized responses even during silence or scene transitions. We conduct extensive evaluations on the REACT2025 benchmark using the MARS dataset. Results show that our method outperforms state-of-the-art models in appropriateness (FRCorr ↑0.71), diversity (FRDiv ↑0.1405), and synchrony (FRSyn ↑47.77), ranking 1st in the offline track and 3rd in the online setting. These findings highlight the framework's effectiveness in simulating human-like, emotionally congruent reactions while offering interpretability grounded in personality psychology.