From Pixels to Semantics: A Novel MLLM-Driven Approach for Explainable Tampered Text Detection

摘要

The spread of tampered text poses a critical challenge to information security. Previous methods for tampered text detection (TTD) primarily relied on visual artifacts as clues, while overlooking potential semantic inconsistencies introduced during text manipulation. To address this limitation, we propose TVSIP (Tampered text Visual-Semantic InterPreter), a novel framework leveraging Multimodal Large Language Models (MLLMs) to integrate both visual and semantic clues for comprehensive tampered text analysis and verification. TVSIP consists of a Locator and an Interpreter. The Locator combines the visual detection ability of existing expert models with the semantic comprehension capabilities of MLLMs to create precise tampering masks. Subsequently, the Interpreter provides comprehensive descriptions and explanations based on identified tampered regions. To train and evaluate TVSIP, we construct the TextDDLE benchmark using GPT-4o. Extensive experiments demonstrate that TVSIP outperforms expert models in pixel-level localization and advanced MLLMs in interpretability. Furthermore, it maintains robustness against image degradation and exhibits strong generalization ability on out-of-domain datasets. Our work highlights the crucial role of semantic inconsistencies in TTD and establishes a more reliable verification system for ensuring document authenticity in the digital age.