Navigating Truth in Multimodal Fact-checking via Retrieval- and Reasoning-Enhanced Large Language Models

摘要

Recent studies show that claims incorporating both text and images spread more effectively than those with text alone, presenting significant challenges for multimodal fact-checking. The rapid development of Multi-modal Large Language Models (MLLMs) has greatly advanced research in this field, enabling stronger performance. However, existing MLLM-based fact-checking methods fail to fully exploit visual evidence, and their reliance on rigid fine-tuning templates limits context-aware explanations and leads to weak deep reasoning. To address these limitations, we propose FACTCOMPASS, a novel framework that combines reasoning-aware fine-tuning with large-scale rule-based reinforcement learning and incorporates a semantic- and knowledge-enhanced retrieval module to strengthen deep reasoning and improve evidence utilization. This framework enhances evidence retrieval by obtaining semantically relevant evidence images, enriching the contextual understanding of claim-related images, and refining textual evidence at the knowledge level. To further enhance reasoning, we introduce a self-refining reinforcement fine-tuning strategy: (1) distilling GPT-4o's reasoning from partially fact-checking data for cold-start Chain-of-Thought learning; (2) activating reasoning across broader datasets using prior knowledge and rejection sampling; (3) applying Group Relative Policy Optimization to explore diverse reasoning paths and optimize factual consistency. Extensive experiments have demonstrated the effectiveness of the proposed framework.