The Ava-MERG Challenge at ACM Multimedia 2025 explores avatar-based multimodal empathetic response generation by advancing dialogue systems that respond with emotional awareness across text, speech, and talking-face video. To support this goal, we constructed a high-quality dataset featuring aligned multimodal responses in diverse conversational scenarios. The challenge comprises two progressively complex subtasks centering around multimodal-aware empathetic response generation. We aim to promote research in empathetic AI and affective computing.
论文检索
输入标题、作者或关键词，从 1,444 篇学术成果中精准定位
已筛选
关键词：Dialogue
1,444篇论文匹配“Dialogue”
第 25 / 73 页
We present a Collaborative Agent-Based Framework for Multi-Image Reasoning. Our approach tackles the challenge of interleaved multimodal reasoning across diverse datasets and task formats by employing a dual-agent system: a language-based PromptEngineer, which generates context-aware, task-specific prompts, and a VisionReasoner, a large vision-language model (LVLM) responsible for final inference. The framework is fully automated, modular, and training-free, enabling generalization across classification, question answering, and free-form generation tasks involving one or multiple input images. We evaluate our method on 18 diverse datasets from the 2025 MIRAGE Challenge (Track A), covering a broad spectrum of visual reasoning tasks including document QA, visual comparison, dialogue-based understanding, and scene-level inference. Our results demonstrate that LVLMs can effectively reason over multiple images when guided by informative prompts. Notably, Claude 3.7 achieves near-ceiling performance on challenging tasks such as TQA (99.13% accuracy), DocVQA (96.87%), and MMCoQA (75.28 ROUGE-L). We also explore how design choices-such as model selection, shot count, and input length-influence the reasoning performance of different LVLMs.
This project presents an interactive installation that integrates artificial intelligence and visual art, employing immersive environments, AI-driven emotional dialogue systems to explore the possibility of emotional interactions between humans and AI in romantic scenarios. Participants enter a virtual space to engage in romantic conversations with an AI virtual character, sharing their emotions. Meanwhile, external observers use VR devices to experience the interaction from the AI's perspective, enabling a philosophical reflection through a third-person viewpoint. This dual-layered experience ultimately unveils a crucial question: In an era where human-machine interactions blur the boundaries of subjectivity, are emotions the product of genuine connections with an ''other,'' or do they merely reflect a narcissistic self-satisfaction shaped by technology? Through artistic expression, this project provokes profound reflections on subjectivity, the meaning of emotions, and the boundaries of human-AI relationships in a posthumanist context, challenging traditional definitions of emotional essence and offering groundbreaking perspectives.
Multimodal hate speech detection targets offensive content expressed through combinations of modalities such as text and images, which often evade detection when analyzed separately. We introduce MAXplain, an interactive framework that addresses both issues via a configurable LLM-based multi-agent architecture. Specialized agents handle distinct subtasks and exchange information through structured dialogues, enabling intrinsic explainability and improved accuracy. The web interface supports human-in-the-loop interaction, including real-time adjustment of agent behaviors and evaluation rules. A browser plugin enables direct inspection of online content. While demonstrated for hate speech detection, MAXplain also supports rapid prototyping for other multimodal tasks.
Visual anomaly detection (VAD) aims to identify image regions that deviate from established normal patterns. Existing methods often rely on domain-specific training and follow a ''one-class-one-model'' paradigm, limiting scalability. We propose Omni-LLaMA-AD, the first unified multimodal large language model for open-set anomaly detection, capable of handling diverse domains with minimal supervision. Built on a pretrained LLaMA backbone, the model uses a VQGAN-based tokenizer and supports joint vision-language generation. Trained via vision-language alignment and instruction tuning, it achieves effective anomaly detection with only a few normal samples and no domain-specific fine-tuning. Our demo showcases the model's ability to generate high-quality anomaly masks across industrial, medical, and logical datasets, highlighting its strong cross-domain generalization and interactive dialogue-based user experience.
Genesis: A Large-Scale Benchmark for Multimodal Large Language Model in Emotional Causality Analysis
Artificial intelligence will not achieve genuine empathy until models can reason about the causes of human emotions rather than only label them. Current datasets fail to support this objective, as existing emotional causality datasets primarily focus on textual modalities, lack non-verbal information such as speech and facial expressions, feature relatively short dialogue lengths, and limit research on long-term emotional evolution. Existing annotations concentrate on stimulus-response patterns and lack cross-temporal emotional causal chain annotations, failing to reveal how early events accumulate and ultimately trigger emotional changes. In this work, we introduce Genesis, the first multimodal dialogue dataset supporting long-term emotional causality analysis, which Genesis contains 1,000 dialogues averaging 208 turns each, spanning debate, family, educational, and social scenarios. Through two-layer annotation system: proximal cause identification and long-term causal chain tracking, Genesis labels complex emotional phenomena including cross-modal inconsistencies and long-distance causal dependencies. Our evaluation of 20 mainstream multimodal models reveals limitations in current approaches for long-term emotional causality. We propose Empathica as an evaluation baseline, employing a Recognition-Memory-Attribution architecture that integrates dynamic sliding windows and event aggregation mechanisms to address multimodal emotional causality modeling challenges. Empathica outperforms text-based models GPT-o1, and multimodal model Gemini 1.5 Pro and GPT-4o across all evaluation metrics.
The integration of multimodal information, particularly visual content, into dialogue systems has primarily focused on interpreting user-provided inputs, while comparatively little attention has been given to the proactive use of such content to enhance responses. In this paper, we explore a new research direction that addresses this gap by enabling dialogue systems to autonomously determine when and how to supplement textual responses with relevant images, based on conversational context and user intent. To support this goal, we propose AMI (Automated Multimodal Insertion), a novel framework for dynamic, context-aware multimodal supplementation in dialogue. We also introduce RID (Response with Appropriate Image Dataset), a bilingual (Chinese-English) multimodal multi-turn dialogue dataset designed to train and evaluate systems on this capability. RID features fine-grained annotations on image insertion timing and rationale, along with carefully aligned image-text pairs to ensure semantic coherence. Our experiments demonstrate that models trained with RID not only generate more informative and engaging responses, but also exhibit a stronger ability to leverage visual content when it is truly beneficial. These findings highlight the potential of proactive multimodal supplementation and offer new insights for advancing the development of intelligent, human-like dialogue systems. Code and data are available at: https://github.com/Tanthen/SCIR-AMI.
Tackling toxic behavior in digital communication continues to be a pressing concern for both academics and industry professionals. While significant research has explored toxicity on platforms like social networks and discussion boards, podcasts-despite their rapid rise in popularity-remain relatively understudied in this context. This work seeks to fill that gap by curating a dataset of political podcast transcripts and analyzing them with a focus on conversational structure. Specifically, we investigate how toxicity surfaces and intensifies through sequences of replies within these dialogues, shedding light on the organic patterns by which harmful language can escalate across conversational turns. Warning: Contains potentially abusive/toxic contents.
Large vision-language models (LVLMs) have recently achieved significant advancements, demonstrating powerful capabilities in understanding and reasoning about visual information. However, LVLMs may generate biased responses that reflect the user beliefs rather than the facts, a phenomenon known as sycophancy. Sycophancy can pose serious challenges to the performance, trustworthiness, and security of LVLMs, raising concerns about their practical applications. We note that there is limited work on the evaluation and mitigation of sycophancy in LVLMs. In this paper, we introduce SyEval-VL, a benchmark specifically designed to evaluate sycophancy in LVLMs. SyEval-VL offers a comprehensive evaluation of sycophancy in visual understanding and reasoning across various scenarios with a multi-round dialogue format. We evaluate sycophancy in several popular LVLMs, providing an in-depth analysis of various sycophantic behaviors and their consequential impacts. Additionally, we propose a novel framework, Human Feedback-based Retrieval-Augmented Generation (HFRAG), to mitigate sycophancy in LVLMs by determining the appropriate timing of retrieval, profiling the proper retrieval target, and augmenting the decoding of LVLMs. Extensive experiments demonstrate that the proposed method significantly mitigates sycophancy in LVLMs without requiring additional training. Our code is available at: https://github.com/immc-lab/SyEval-VL
Conversational Speech Synthesis (CSS) is a key task in the user-agent interaction area, aiming to generate more expressive and empathetic speech for users. However, it is well-known that ''listening'' and ''eye contact'' play crucial roles in conveying emotions during real-world interpersonal communication. Existing CSS research is limited to perceiving only text and speech within the dialogue context, which restricts its effectiveness. Moreover, speech-only responses further constrain the interactive experience. To address these limitations, we introduce a Conversational Speech-Visual Synthesis (CSVS) task as an extension of traditional CSS. By leveraging multimodal dialogue context, it provides users with coherent audiovisual responses. To this end, we develop a CSVS system named UniTalker, which is a unified model that seamlessly integrates multimodal perception and multimodal rendering capabilities. Specifically, it leverages a large-scale language model to comprehensively understand multimodal cues in the dialogue context, including speaker, text, speech, and the talking-face animations. After that, it employs multi-task sequence prediction to first infer the target utterance's emotion and then generate empathetic speech and natural talking-face animations. To ensure that the generated speech-visual content remains consistent in terms of emotion, content, and duration, we introduce three key optimizations: 1) Designing a specialized neural landmark codec to tokenize and reconstruct facial expression sequences. 2) Proposing a bimodal speech-visual hard alignment decoding strategy. 3) Applying emotion-guided rendering during the generation stage. Comprehensive objective and subjective experiments demonstrate that our model synthesizes more empathetic speech and provides users with more natural and emotionally consistent talking-face animations. The source code and generated samples are available at: https://github.com/AI-S2-Lab/UniTalker.
Text-to-image person retrieval aims to identify target person images using natural language descriptions. Current state-of-the-art methods predominantly rely on single-round retrieval frameworks, where retrieval accuracy heavily depends on the quality of the initial textual descriptions. However, users sometimes struggle to provide detailed and distinctive descriptions in a single attempt, resulting in generic initial queries that lack discriminative details. This fundamental limitation of the single-round retrieval framework frequently leads to the misinterpretation of user intent and suboptimal retrieval performance. To address this limitation, we propose Dialogue-driven Interactive Dynamic Learning (DIDL) for text-to-image person retrieval. Specifically, we first introduce Collaborative Query Refinement (CQR), which progressively refines retrieval conditions through multi-round dialogues. Then, we design Dynamic Context Resampling (DCR) based on a bi-granular mask strategy that enhances the model's adaptation to dialogue-style contexts and effectively balances its attention between initial descriptions and supplementary information. Based on these components, we further propose cross-modal Probabilistic Context Matching Modeling (ProCMM) that establishes effective associations between static visual features and dynamic contextual semantics. Extensive experiments demonstrate that our approach achieves state-of-the-art performance across all three benchmark datasets.
Previous multimodal emotion-cause analysis in conversations (MEC-AC) has predominantly focused on English, overlooking the applicability of existing methods in multilingual contexts. To bridge this gap, we construct a Chinese contextual dataset (MEC4) to investigate how language and culture diversity influences existing MECAC approaches. Moreover, prior studies often rely on average pooling or frame sampling to extract visual and acoustic features from video and audio of long dialogues, which inevitably results in the loss of temporal dynamics and emotionally salient cues. To overcome these limitations, we propose a memory-inspired multilingual multimodal framework (M3F) based on large language model (LLM), which can effectively capture the temporal and global informative features of non-linguistic modalities through memory bank module. This module simulates the way memory is stored in human cognitive processes and incrementally aggregates past visual and acoustic features in an autoregressive manner, enabling effective reference during future sequence modeling. Through rigorous experiments and insightful analyses, we find that cultural differences cause variations in how emotional expressions in English and Chinese languages rely on modalities.
The automatic generation of diverse and human-like facial reactions in dyadic dialogue remains a critical challenge for human-computer interaction systems. Existing methods fail to model the stochasticity and dynamics inherent in real human reactions. To address this, we propose ReactDiff, a novel temporal diffusion framework for generating diverse facial reactions that are appropriate for responding to any given dialogue context. Our key insight is that plausible human reactions demonstrate smoothness, and coherence over time, and conform to constraints imposed by human facial anatomy. To achieve this, ReactDiff incorporates two vital priors (spatio-temporal facial kinematics) into the diffusion process: i) temporal facial behavioral kinematics and ii) facial action unit dependencies. These two constraints guide the model toward realistic human reaction manifolds, avoiding visually unrealistic jitters, unstable transitions, unnatural expressions, and other artifacts. Extensive experiments on the REACT2024 dataset demonstrate that our approach not only achieves state-of-the-art reaction quality but also excels in diversity and reaction appropriateness. Our code is publicly available at https://github.com/lingjivoo/ReactDiff.
Modern text-to-image generation systems have enabled the creation of remarkably realistic and high-quality visuals, yet they often falter when handling the inherent ambiguities in user prompts. In this work, we present Twin-Co, a framework that leverages synchronized, co-adaptive dialogue to progressively refine image generation. Instead of a static generation process, Twin-Co employs a dynamic, iterative workflow where an intelligent dialogue agent continuously interacts with the user. Initially, a base image is generated from the user's prompt. Then, through a series of synchronized dialogue exchanges, the system adapts and optimizes the image according to evolving user feedback. The co-adaptive process allows the system to progressively narrow down ambiguities and better align with user intent. Experiments demonstrate that Twin-Co not only enhances user experience by reducing trial-and-error iterations but also improves the quality of the generated images, streamlining creative process across various applications.
Conversational recommender systems (CRS) aim to provide personalized recommendations via interactive dialogues with users. While large language models (LLMs) enhance CRS with their superior understanding of context-aware user preferences, they typically struggle to leverage behavioral data, which have proven to be important for classical collaborative filtering (CF)-based approaches. For this reason, we propose CRAG-Collaborative Retrieval Augmented Generation for LLM-based CRS. To the best of our knowledge, CRAG is the first approach that combines state-of-the-art LLMs with CF for conversational recommendations. Our experiments on two publicly available movie conversational recommendation datasets, i.e., a refined Reddit dataset (which we name Reddit-v2) as well as the Redial dataset, demonstrate the superior item coverage and recommendation performance of CRAG, compared to several CRS baselines. Moreover, we observe that the improvements are mainly due to better recommendation accuracy on recently released movies. The code and data are available at https://github.com/yaochenzhu/CRAG.
In-Group Love, Out-Group Hate: A Framework to Measure Affective Polarization via Contentious Online Discussions
PDF ↗Affective polarization, the emotional divide between ideological groups marked by in-group love and out-group hate, has intensified in the United States, driving contentious issues like masking and lockdowns during the COVID-19 pandemic. Despite its societal impact, existing models of opinion change fail to account for emotional dynamics nor offer methods to quantify affective polarization robustly and in real-time. In this paper, we introduce a discrete choice model that captures decision-making within affectively polarized social networks and propose a statistical inference method estimate key parameters---in-group love and out-group hate---from social media data. Through empirical validation from online discussions about the COVID-19 pandemic, we demonstrate that our approach accurately captures real-world polarization dynamics and explains the rapid emergence of a partisan gap in attitudes towards masking and lockdowns. This framework allows for tracking affective polarization across contentious issues has broad implications for fostering constructive online dialogues in digital spaces.
One of the most challenging aspects of learning a foreign language is learning to converse in that language. While large language models (LLMs) have made it feasible to build generative AI chatbots that simulate conversation, creating an AI-powered language tutor that is both pedagogically effective and engaging for learners requires solving a host of additional problems. At Duolingo, we have developed an AI conversational tutor embedded in our platform, featuring a character named Lily, who helps users practice real-world conversations in their target language. In building this system, we tackled multiple challenges: adapting dialogue to each learner's proficiency level, sustaining personalized and coherent interactions across sessions, maintaining consistent character-driven personality, and designing a structure that supports both guided and learner-initiated topics. Our solution integrates a three-party conversational architecture, a persistent memory mechanism to retain prior interactions, and real-time conversation evaluation to dynamically adjust to the learner's input. This talk will highlight how generative AI, when coupled with rigorous feedback loops and thoughtful design, can significantly enhance the language learning experience.
On-demand ride-hailing services like DiDi, Uber, and Lyft have transformed urban transportation, offering unmatched convenience and flexibility. In this paper, we introduce DiMA, an LLM-powered ride-hailing assistant deployed in DiDi Chuxing. Its goal is to provide seamless ride-hailing services and beyond through a natural and efficient conversational interface under dynamic and complex spatiotemporal urban contexts. To achieve this, we propose a spatiotemporal-aware order planning module that leverages external tools for precise spatiotemporal reasoning and progressive order planning. Additionally, we develop a cost-effective dialogue system that integrates multi-type dialog repliers with cost-aware LLM configurations to handle diverse conversation goals and trade-off response quality and latency. Furthermore, we introduce a continual fine-tuning scheme that utilizes real-world interactions and simulated dialogues to align the assistant's behavior with human prefered decision-making processes. Since its deployment in the DiDi application, DiMA has demonstrated exceptional performance, achieving 93% accuracy in order planning and 92% in response generation during real-world interactions. Offline experiments further validate DiMA's capabilities, showing improvements of up to 70.23% in order planning and 321.27% in response generation compared to three state-of-the-art agent frameworks, while reducing latency by 0.72x to 5.47x. These results establish DiMA as an effective, efficient, and intelligent mobile assistant for ride-hailing services.
An emerging type of fraud involves malicious senders exploiting the blind shipment and cash-on-delivery (COD) mechanisms by dispatching large volumes of unsolicited, low-cost parcels. If unsuspecting receivers accept these parcels, they pay for both shipping and goods; otherwise, logistics providers bear the round-trip shipping costs. Existing detection techniques, which rely on extensive labeled cases, struggle with this emerging fraud because receivers' unawareness and low transaction values discourage complaints, resulting in few confirmed cases. Therefore, we propose leveraging receivers' complaints, though not initially collected for fraud detection, to uncover subtle indicators of fraud patterns, while addressing three challenges: (C1) noise-rich dialogues(C2) data privacy concerns, and (C3) ever-evolving fraud patterns. To address them, we design BLOFF, a Blind shipment detection Framework for LO gistics Fraud powered by large language models (LLMs). Specifically, BLOFF includes three components: i) Sensitivity Anonymization to protect sensitive user information; ii) Dialogue Profile Distillation to transform informal dialogues into structured representation, addressing C1, and distill knowledge from a teacher LLM (GPT-4o) to a lightweight student LLM (ChatGLM4-9B), addressing C2; ii) Multi-faceted Context Augmentation to enhance the interpretation of fraud signatures and adaptation of evolving patterns, addressing C3. We evaluate BLOFF on about 56,000 complaints records collected from JD Logistics between January and November 2024. Results show that BLOFF outperforms state-of-the-art methods, achieving a 10.19% improvement in precision. Furthermore, during its real-world deployment in December 2024, BLOFF identified over 90 fraudulent parcels with a 91.4% precision.
Online recruitment platforms have reshaped job-seeking and recruiting processes, driving increased demand for applications that enhance person-job matching. Traditional methods generally rely on analyzing textual data from resumes and job descriptions, limiting the dynamic, interactive aspects crucial to effective recruitment. Recent advances in Large Language Models (LLMs) have revealed remarkable potential in simulating adaptive, role-based dialogues, making them well-suited for recruitment scenarios. In this paper, we propose MockLLM, a novel framework to generate and evaluate mock interview interactions. The system consists of two key components: mock interview generation and two-sided evaluation in handshake protocol. By simulating both interviewer and candidate roles, MockLLM enables consistent and collaborative interactions for real-time and two-sided matching. To further improve the matching quality, MockLLM further incorporates reflection memory generation and dynamic strategy modification, refining behaviors based on previous experience. We evaluate MockLLM on real-world data Boss Zhipin, a major Chinese recruitment platform. The experimental results indicate that MockLLM outperforms existing methods in matching accuracy, scalability, and adaptability across job domains, highlighting its potential to advance candidate assessment and online recruitment.