DeepSIX at ACM MM 2025 Grand Challenge: Enhancing Context Text Processing for Multimodal Hallucination Detection and Fact Verification

摘要

Significant advancements have been achieved in both fields of Natural Language Processing (NLP) and Computer Vision (CV) with the advent of Multimodal Large Language Models (MLLMs), sometimes referred to as large vision-language models (LVMs). MLLMs show promising ability in multimodal tasks, such as image captioning, visual question answering, etc. However, there is a concerning trend associated with the advancement in MLLMs. These models exhibit an inclination to generate hallucinations and misleading facts, resulting in seemingly plausible yet factually spurious content. To address these challenges, our team, DeepSIX, leverages recent advances in MLLMs to enhance the ability to detect hallucination and verify factual information within the scope of the ACM MM 2025 grand challenge 8: Truthful and Responsible Multimodal Learning (ResMM). We participated in both tasks: Multimodal Hallucination Detection (Task 1) and Multimodal Fact Checking (Task 2). Our approach leverages the interpretive power of the vision and language components of vision language models (VLMs) to analyze and summarize insights from text and images. It performs contextual reasoning by uncovering semantic relationships among entities in the text and objects in the images. By employing diverse prompting techniques, our method deconstructs critical entities in the text, effectively uncovers implicit relationships between text and images, and identifies hallucinations and false facts. Experimental results demonstrate the strength of our approach: it achieved second place in the Hallucination Detection task and third place in the Fact Verification task, confirming the potential of LLM-based methods in MLLMs. We open-source our code at https://github.com/JAIST-DeepSIX/ACMMM25