论文检索

Rolandos Alexandros Potamias, Stathis Galanakis, Jiankang Deng, Athanasios Papaioannou, Stefanos Zafeiriou

Over the last years, 3D morphable models (3DMMs) have emerged as a state-of-the-art methodology for modeling and generating expressive 3D avatars. However, given their reliance on a strict topology, along with their linear nature, they struggle to represent complex full-head shapes. Following the advent of deep implicit functions, we propose imHead, a novel implicit 3DMM that not only models expressive 3D head avatars but also facilitates localized editing of the facial features. Previous methods directly divided the latent space into local components accompanied by an identity encoding to capture the global shape variations, leading to expensive latent sizes. In contrast, we retain a single compact identity space and introduce an intermediate region-specific latent representation to enable local edits. To train imHead, we curate a large-scale dataset of 4K distinct identities, making a step-towards large scale 3D head modeling. Under a series of experiments we demonstrate the expressive power of the proposed model to represent diverse identities and expressions outperforming previous approaches. Additionally, the proposed approach provides an interpretable solution for 3D face manipulation, allowing the user to make localized edits.

Dimitrios Mallis, Ahmet Serda Karadeniz, Sebastian Cavada, Danila Rukhovich, Niki Foteinopoulou, Kseniya Cherenkova, Anis Kacem, Djamila Aouada

We propose CAD-Assistant, a general-purpose CAD agent for AI-assisted design. Our approach is based on a powerful Vision and Large Language Model (VLLM) as a planner and a tool-augmentation paradigm using CAD-specific tools. CAD-Assistant addresses multimodal user queries by generating actions that are iteratively executed on a Python interpreter equipped with the FreeCAD software, accessed via its Python API. Our framework is able to assess the impact of generated CAD commands on geometry and adapts subsequent actions based on the evolving state of the CAD design. We consider a wide range of CAD-specific tools including a sketch image parameterizer, rendering modules, a 2D cross-section generator, and other specialized routines. CAD-Assistant is evaluated on multiple CAD benchmarks, where it outperforms VLLM baselines and supervised task-specific methods. Beyond existing benchmarks, we qualitatively demonstrate the potential of tool-augmented VLLMs as general-purpose CAD solvers across diverse workflows.

Xiaohui Chen, Satya Narayan Shukla, Mahmoud Azab, Aashu Singh, Qifan Wang, David Yang, ShengYun Peng, Hanchao Yu, Shen Yan, Xuewen Zhang 等

How well can Multimodal Large Language Models (MLLMs) understand composite images? Composite images (CIs) are synthetic visuals created by merging multiple visual elements, such as charts, posters, or screenshots, rather than being captured directly by a camera. While CIs are prevalent in real-world applications, recent MLLM developments have primarily focused on interpreting natural images (NIs). Our research reveals that current MLLMs face significant challenges in accurately understanding CIs, often struggling to extract information or perform complex reasoning based on these images. We find that existing training data for CIs are mostly formatted for question-answer tasks (e.g., in datasets like ChartQA and ScienceQA), while high-quality image-caption datasets, critical for robust vision-language alignment, are only available for NIs. To bridge this gap, we introduce Composite Captions (CompCap), a flexible framework that leverages Large Language Models (LLMs) and automation tools to synthesize CIs with accurate and detailed captions. Using CompCap, we curate CompCap-118K, a dataset containing 118K image-caption pairs across six CI types. We validate the effectiveness of CompCap-118K by supervised fine-tuning MLLMs of three sizes: xGen-MM-inst.-4B and LLaVA-NeXT-Vicuna-7B/13B. Empirical results show that CompCap-118K significantly enhances MLLMs' understanding of CIs, yielding average gains of 1.7%, 2.0%, and 2.9% across eleven benchmarks, respectively.

Shijie Fang, Hongping Gan

Deep Unfolding Networks (DUNs) have emerged as a powerful framework for pansharpening due to their interpretable fusion strategies. However, existing DUNs are limited by their serial iterative architectures, which hinder cross-stage and cross-modal feature interactions at different abstraction levels. This limitation results in insufficient integration of multi-level multimodal features and compromised reconstruction accuracy. To address these challenges, we propose the Unfolding-Associative Encoder-Decoder Network (UED-Net), an innovative framework that iteratively extracts multi-level cross-modal degradation encodings and recursively refines features for cross-stage adaptive aggregation decoding through lightweight processes. Specifically, we first introduce the spatial-spectral encoding module, which progressively and interpretably perceives the hierarchical degradation encoding features of both space and spectrum. Moreover, we develop the unfolding-associative attention module to capture pixel-level attention across stages, thereby leveraging the causal relationships of multi-level features for aggregation during decoding. Meanwhile, we implement a progressive alignment mechanism, which coordinates both feature distribution and alignment of spatial and spectral modalities between iterative stages to facilitate adaptive fusion. These modules enable UED-Net to achieve efficient pansharpening by aggregating multi-level features. Extensive qualitative and quantitative experiments confirm the superiority of UED-Net.

Dahye Kim, Xavier Thomas, Deepti Ghadiyaram

We study how rich visual semantic information is represented within various layers and denoising timesteps of different diffusion architectures. We uncover monosemantic interpretable features by leveraging k-sparse autoencoders (k-SAE). We substantiate our mechanistic interpretations via transfer learning using light-weight classifiers on off-the-shelf diffusion models' features. On 4 datasets, we demonstrate the effectiveness of diffusion features for representation learning. We provide an in-depth analysis of how different diffusion architectures, pre-training datasets, and language model conditioning impacts visual representation granularity, inductive biases, and transfer learning capabilities. Our work is a critical step towards deepening interpretability of black-box diffusion models. Code and visualizations available at: https://github.com/revelio-diffusion/revelio

Fatemeh Ghezloo, Mehmet Saygin Seyfioglu, Rustin Soraki, Wisdom O. Ikezogwo, Beibin Li, Tejoram Vivekanandan, Joann G. Elmore, Ranjay Krishna, Linda Shapiro

Diagnosing diseases through histopathology whole slide images (WSIs) is fundamental in modern pathology but is challenged by the gigapixel scale and complexity of WSIs. Trained histopathologists overcome this challenge by navigating the WSI, looking for relevant patches, taking notes, and compiling them to produce a final holistic diagnostic. Traditional AI approaches, such as multiple instance learning and transformer-based models, fail short of such a holistic, iterative, multi-scale diagnostic procedure, limiting their adoption in the real-world. We introduce PathFinder, a multi-modal, multi-agent framework that emulates the decision-making process of expert pathologists. PathFinder integrates four AI agents, the Triage Agent, Navigation Agent, Description Agent, and Diagnosis Agent, that collaboratively navigate WSIs, gather evidence, and provide comprehensive diagnoses with natural language explanations. The Triage Agent classifies the WSI as benign or risky; if risky, the Navigation and Description Agents iteratively focus on significant regions, generating importance maps and descriptive insights of sampled patches. Finally, the Diagnosis Agent synthesizes the findings to determine the patient's diagnostic classification. Our Experiments show that PathFinder outperforms state-of-the-art methods in skin melanoma diagnosis by 8% while offering inherent explainability through natural language descriptions of diagnostically relevant patches. Qualitative analysis by pathologists shows that the Description Agent's outputs are of high quality and comparable to GPT-4o. PathFinder is also the first AI-based system to surpass the average performance of pathologists in this challenging melanoma classification task by 9%, setting a new record for efficient, accurate, and interpretable AI-assisted diagnostics in pathology. Our Data, code and models will be made available.

Zixian Guo, Ming Liu, Qilong Wang, Zhilong Ji, Jinfeng Bai, Lei Zhang, Wangmeng Zuo

Current large vision-language models (LVLMs) typically employ a connector module to link visual features with text embeddings of large language models (LLMs) and use end-to-end training to achieve multi-modal understanding in a unified process. Well alignment needs high-quality pre-training data and a carefully designed training process. Current LVLMs face challenges when addressing complex vision-language reasoning tasks, with their reasoning capabilities notably lagging behind those of LLMs. This paper proposes a paradigm shift: instead of training end-to-end vision-language reasoning models, we advocate for developing a decoupled reasoning framework based on existing visual interpretation specialists and text-based reasoning LLMs. Our approach leverages (1) a dedicated vision-language model to transform the visual content of images into textual descriptions and (2) an LLM to perform reasoning according to the visual-derived text and the original question. This method presents a cost-efficient solution for multi-modal model development by optimizing existing models to work collaboratively, avoiding end-to-end development of vision-language models from scratch. By transforming images into language model-compatible text representations, it facilitates future low-cost and flexible upgrades to upcoming powerful LLMs. We introduce an outcome-rewarded joint-tuning strategy to optimize the cooperation between the visual interpretation and linguistic reasoning model. Evaluation results on vision-language benchmarks demonstrate that the decoupled reasoning framework outperforms recent LVLMs. Our approach yields particularly significant performance gains on visually intensive geometric mathematics problems. The code is available: https://github.com/guozix/DVLR.

Xin Qiao, Matteo Poggi, Xing Wei, Pengchao Deng, Yanhui Zhou, Stefano Mattoccia

Under-display ToF imaging aims to achieve accurate depth sensing through a ToF camera placed beneath a screen panel. However, transparent OLED (TOLED) layers introduce severe degradations--such as signal attenuation, multi-path interference (MPI), and temporal noise--that significantly compromise depth quality. To alleviate this drawback, we propose Learnable Fractional Reaction-Diffusion Dynamics (LFRD^2), a hybrid framework that combines the expressive power of neural networks with the interpretability of physical modeling. Specifically, we implement a time-fractional reaction-diffusion module that enables iterative depth refinement with dynamically generated differential orders, capturing long-term dependencies. In addition, we introduce an efficient continuous convolution operator via coefficient prediction and repeated differentiation to further improve restoration quality. Experiments on four benchmark datasets demonstrate the effectiveness of our approach. The code is publicly available at https://github.com/wudiqx106/LFRD2.

Michihiro Kuroki, Toshihiko Yamasaki

Although saliency maps can highlight important regions to explain the reasoning behind image classification in artificial intelligence (AI), the meaning of these regions is left to the user's interpretation. In contrast, concept-based explanations decompose AI predictions into human-understandable concepts, clarifying their contributions. However, few methods can simultaneously reveal what concepts an image classifier learns, which regions are associated with them, and how they contribute to predictions.We propose a novel concept-based explanation method, Concept-based Explanation via Fusion of Activation Maps (CE-FAM). It employs a branched network that shares activation maps with an image classifier and learns to mimic the embeddings of a Vision and Language Model (VLM). The branch network predicts concepts in an image, and their corresponding regions are represented by a weighted sum of activation maps, with weights given by the gradients of the concept prediction scores. Their contributions are quantified based on their impact on the image classification score. Our method provides a general framework for identifying the concept regions and their contributions while leveraging VLM knowledge to handle arbitrary concepts without requiring an annotated dataset. Furthermore, we introduce a novel evaluation metric to assess the accuracy of the concept regions. Our qualitative and quantitative evaluations demonstrate our method outperforms existing approaches and excels in zero-shot inference for unseen concepts.

Yang Xiao, Wang Lu, Jie Ji, Ruimeng Ye, Gen Li, Xiaolong Ma, Bo Hui

The design of artificial neural networks (ANNs) is inspired by the structure of the human brain, and in turn, ANNs offer a potential means to interpret and understand brain signals. Existing methods primarily align brain signals with stimulus signals using Mean Squared Error (MSE), which focuses only on local point-wise alignment and ignores global matching, leading to coarse interpretations and inaccuracies in brain signal decoding. In this paper, we address these issues through optimal transport (OT) and theoretically demonstrate why OT provides a more effective alignment strategy than MSE. Specifically, we construct a transport plan between brain voxel embeddings and image embeddings, enabling more precise matching. By controlling the amount of transport, we mitigate the influence of redundant information. We apply our alignment model directly to the Brain Captioning task by feeding brain signals into a large language model (LLM) instead of images. Our approach achieves state-of-the-art performance across ten evaluation metrics, surpassing the previous best method by an average of 6.11% in singlesubject training and 3.81% in cross-subject training. Additionally, we have uncovered several insightful conclusions that align with existing brain research. We unveil the redundancy and synergy of brain information processing through region masking and data dimensionality reduction visualization experiments. We believe our approach paves the way for a more precise understanding of brain signals in the future. The code is available at https://github.com/NKUShaw/OT-Alignment4brain-to-image

Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, Zhaoxiang Zhang

The rapid development of Large Multimodal Models (LMMs) for 2D images and videos has spurred efforts to adapt these models for interpreting 3D scenes. However, the absence of large-scale 3D vision-language datasets has posed a significant obstacle. To address this issue, typical approaches focus on injecting 3D awareness into 2D LMMs by designing 3D input-level scene representations. This work provides a new perspective. We introduce reconstructive visual instruction tuning with 3D-awareness (ROSS3D), which integrates 3D-aware visual supervision into the training procedure. Specifically, it incorporates cross-view and global-view reconstruction. The former requires reconstructing masked views by aggregating overlapping information from other views. The latter aims to aggregate information from all available views to recover Bird's-Eye-View images, contributing to a comprehensive overview of the entire scene. Empirically, ROSS3D achieves state-of-the-art performance across various 3D scene understanding benchmarks. More importantly, our semi-supervised experiments demonstrate significant potential in leveraging large amounts of unlabeled 3D vision-only data.

Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, Xiang Bai

Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model, enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at https://github.com/LMD0311/HERMES.

Chirui Chang, Jiahui Liu, Zhengzhe Liu, Xiaoyang Lyu, Yi-Hua Huang, Xin Tao, Pengfei Wan, Di Zhang, Xiaojuan Qi

Recent advancements in video diffusion models enable the generation of photorealistic videos with impressive 3D consistency and temporal coherence. However, the extent to which these AI-generated videos simulate the 3D visual world remains underexplored. In this paper, we introduce Learned 3D Evaluation (L3DE), an objective, quantifiable, and interpretable method for assessing AI-generated videos' ability to simulate the real world in terms of 3D visual qualities and consistencies, without requiring manually labeled defects or quality annotations. Instead of relying on 3D reconstruction, which is prone to failure with in-the-wild videos, L3DE employs a 3D convolutional network, trained on monocular 3D cues of motion, depth, and appearance, to distinguish real from synthetic videos. Confidence scores from L3DE quantify the gap between real and synthetic videos in terms of 3D visual coherence, while a gradient-based visualization pinpoints unrealistic regions, improving interpretability. We validate L3DE through extensive experiments, demonstrating strong alignment with 3D reconstruction quality and human judgments. Our evaluations on leading generative models (e.g., Sora, MiniMax, and Kling) reveal persistent simulation gaps and subtle inconsistencies. Beyond generative video assessment, L3DE extends to broader applications: benchmarking video generation models, serving as a deepfake detector, and enhancing video synthesis by inpainting flagged inconsistencies.

Ting Yao 0003

In the wave of Artificial Intelligence, along with the proliferation of mobile devices, digital content has been generated and published in an explosive way. Digital content is inherent multimodal: text, image, audio, video, etc. How to effectively and efficiently automate the entire multimodal content lifecycle from idea, to creation and distribution is therefore of great importance. In this talk, we will first delve into the core technological breakthroughs in multimodal content generation, particularly the latest advancements in image and video generation tasks. Then, we will present an agent-based solution from HiDream.ai to interpret user intention and manage content creation-from ideation to final distribution-using three core, interconnected agents. In between, the Content Creation Agent takes a simple user prompt, instantly comprehends the creator's intention, and accurately sources relevant assets from content platform to create multimodal content. Such way frees creators to focus on the story, not the software. The Self-evolving Platform Agent translates global trends and user preferences into strategic directives, guiding models to autonomously generate high-impact content and continuously enriching the platform's ecosystem. The Distribution Agent adapts multimodal content for various social media platforms and analyzes its performance after publishing. Together, these intelligent agents create a seamless ecosystem to connect creators, content and consumers.

Benoit Huet

The integration of AI/ML technologies into medical imaging is revolutionizing radiology, offering transformative benefits in clinical workflows. AI-powered Software as a Medical Device (SaMD) solutions not only reduce workload and optimize image interpretation but also unlock critical insights previously undetectable by human eyes-catching the unseen and enabling earlier, more accurate diagnoses. Lung cancer, the leading cause of cancer-related mortality worldwide, is often diagnosed at a late stage, when curative treatment is no longer viable. Early detection is paramount. Traditional screening methods rely heavily on nodule size and growth as indicators of malignancy. However, these criteria alone are insufficient for identifying cancer at its earliest, most treatable stage. eyonis® LCS, the flagship clinical development program of Median Technologies, represents a next-generation AI/ML-based SaMD designed specifically for lung cancer screening [1][2]. It combines Computer-Aided Detection (CADe) and Computer-Aided Diagnosis (CADx) [3][4] capabilities to support clinicians in identifying malignant nodules with greater precision. By leveraging specific architectural choices and deep learning models, eyonis® LCS enhances diagnostic accuracy beyond the current standard of care [5], offering a paradigm shift in early lung cancer detection [6]. This presentation will delve into some of the architectural foundations of eyonis® LCS, highlight its clinical impact, and demonstrate how it empowers radiologists to diagnose lung cancer when patients can still be cured. Through this pioneering technology, Median Technologies is redefining the future of cancer screening and patient outcomes.

Aleksandr Farseev

In this industry demonstration, we present SOMIN - an Explainable AI and LLM Platform for Real-Time, Data-Driven Digital Marketing Strategy recommendation. This is achieved through two primary, interconnected subsystems: a predictive model for performance analysis and an LLM for semantic understanding. The system is built on SoWide-ViT, a ''wide and deep'' neural network for advertiser-side CTR prediction. It processes tabular features (campaign settings), text (ad copy, headlines), and visuals (images or keyframes) through TabTransformer, multilingual BERT, and a Vision Transformer (ViT). Replacing the previous ABN model with ViT improved performance, reaching an F1-score of 0.78. Explainability comes from ViT's self-attention, which produces heatmaps highlighting influential regions. High-performing ads showed relevant cues (e.g., gaming objects), while low-performing ones exposed distracting elements. Heatmaps show where the model focuses, while the LLM explains why and suggests improvements. A multimodal GPT-5 interprets creatives, finding flaws such as weak hierarchy, poor CTAs, or off-brand imagery. At scale, the Content Library and Perspective Studies classify competitor ads into 16 marketing concepts, clustering them into Personas (e.g., ''Chocolate Connoisseur'') and Tensions (e.g., ''Work Stress''). The SOINSPIRE module converts Personas and Tensions into insights and campaign propositions, applying marketing theories to generate repeatable, data-driven Expressions.

Zheng Lian 0004, Shreya Ghosh 0001, Erik Cambria, Zhixi Cai, Guoying Zhao 0001, Abhinav Dhall, Björn W. Schuller, Roland Goecke, Jianhua Tao 0001, Tom Gedeon

Multimodal, generative, and responsible affective computing aims to enhance people's lives. In recent years, the AI revolution has already begun to impact daily life, with virtual assistants being deployed across various sectors such as healthcare, banking, transportation, and education. It is clear that, in the near future, humans may interact with AI-powered systems as much or maybe even more than direct human-to-human interactions. Affective computing has numerous applications, including innovative approaches to forecasting and preventing anxiety, stress, and mental health issues; enhancing robotic empathy; assisting individuals with communication, behavior, and emotion regulation challenges; and promoting awareness of health and well-being. Many of these applications require enhanced control and protection of sensitive, private, and personal data. Therefore, it is crucial to further develop the creation, evaluation, and deployment of emotionally intelligent systems that are both responsive and responsible. Additionally, improving the accuracy and interpretability of emotion prediction results can significantly enhance the application of this technology in the downstream tasks mentioned above. MRAC'25 is the continuation of MRAC'23 and MRAC'24. Through this workshop, we aim to bring together researchers to discuss the potential and development of affective computing.

Deyuan Chen, Xiaocui Yang, Shi Feng 0001, Zihan Cheng, Daling Wang, Yifei Zhang 0003

In recent years, Multimodal Sentiment Analysis (MSA) has attracted growing attention for its ability to interpret human emotions by integrating information across multiple modalities. Multimodal Conversational Aspect-based Sentiment Analysis (MCABSA) extends this research frontier by incorporating multi-party conversational contexts and requiring comprehensive extraction of sentiment elements. MCABSA presents substantial challenges, including the need to understand complex conversational contexts, integrate heterogeneous multimodal signals, and identify causal reasoning at the cognitive level. To address these challenges, we propose a two-stage Full Fine-tuning and LLM Post-processing (FLP) framework. In the first stage, we develop a multimodal caption-enhanced full fine-tuning pipeline that performs structured extraction of sextuples and sentiment flip tuples. The second stage introduces paraphrase-based sextuple verification to identify and filter low-quality sextuples for Panoptic Sentiment Sextuple Extraction (Task-1), while implementing trigger classification with a distribution alignment mechanism to determine trigger types for sentiment flipping and enhance output consistency for Sentiment Flipping Analysis (Task-2). Comprehensive experiments on both MCABSA challenge subtasks demonstrate the effectiveness of our approach, achieving 1st place on Task-1 and 3rd place on Task-2.

Peng Wang 0210, Pujun Xue, Xiaofeng Liu 0006, Tongjuan Ji

Generating diverse and contextually appropriate facial reactions remains a significant challenge due to variability in individual responses, limited explainability, and insufficient modeling of contextual cues. In this study, we propose a multimodal framework that integrates behavioral memory, dynamic attention control, and cognitive style modeling to generate personalized and psychologically grounded facial reactions in dyadic interactions. Our method models the causal link between speaker behavior and listener response by incorporating frame-level behavioral cues, personality traits, and cognitive processing styles. The proposed system consists of three core components: a behavioral memory module that captures temporal context across conversation turns; a Personalized Personality Recognition Style (PPRS) module that infers cognitive tendencies via dual-path learning based on the Big Five personality traits; and a transformer-based generative module equipped with diffusion modeling and context-aware attention gating. This design enables the generation of expressive, individualized responses even during silence or scene transitions. We conduct extensive evaluations on the REACT2025 benchmark using the MARS dataset. Results show that our method outperforms state-of-the-art models in appropriateness (FRCorr ↑0.71), diversity (FRDiv ↑0.1405), and synchrony (FRSyn ↑47.77), ranking 1st in the offline track and 3rd in the online setting. These findings highlight the framework's effectiveness in simulating human-like, emotionally congruent reactions while offering interpretability grounded in personality psychology.

Yujing Wang, Ruotong Fang, Xing Huang, Zhiyuan Han, Xiaoqing Lin, Yuhao Shan, Tong Chen 0008

This paper presents our solution for the Micro-Expression Visual Question Answering (ME-VQA) task in the 2025 Facial Micro-Expression Grand Challenge (MEGC). To address the limitations of traditional micro-expression recognition (MER) methods in dynamic modeling, semantic interpretation, and natural language interaction, we propose Emotion-Qwen-VL, a fully fine-tuned multimodal large language model tailored for micro-expression understanding. Specifically, we construct a structured, instruction-based QA dataset that reformulates emotion categories and action unit (AU) annotations into natural language QA pairs, covering classification, AU detection, and causal reasoning. We then adopt a full-parameter fine-tuning strategy to guide Qwen2.5-VL in learning fine-grained temporal facial dynamics and their emotional semantics. Experimental results on the MEGC 2025 test set demonstrate that Emotion-Qwen-VL outperforms strong baselines such as Qwen2.5-VL and QVQ across multiple dimensions, including coarse-grained macro-expression classification, fine-grained micro-expression classification, and language generation. Our results highlight the effectiveness, interpretability, and adaptation potential of large models in micro-expression understanding. The code is available at: https://github.com/2308623956/MEGC2025.