Unified Dual-Strategy Framework for Multi-Task Visual Question Answering

摘要

We present a unified vision-language framework for the Responsible Multimodal AI Challenge 2025, tackling two related tasks: multimodal hallucination detection and factuality verification. Our approach builds on a Dynamic ViLBERT model enhanced with adaptive multi-branch attention fusion to effectively integrate visual and textual data. For Task A (Hallucination Detection), we employ four parallel answer-encoding branches with co-attentional transformers to compare AI-generated captions or answers against corresponding images, enabling accurate hallucination identification. For Task B (Factuality Verification), we fuse visual and textual features via element-wise addition, multiplication, and concatenation, feeding them into a classifier to assess claim validity. Inputs include visual features from Detectron2 and text embeddings from BERT tokenization. Experiments show notable accuracy gains over baselines, validated by official F1 scores. Extensive ablations, comparative studies, and architectural visualizations confirm the value of cross-modal attention and customized fusion strategies. Our final system ranks 3rd in hallucination detection (F1 = 0.80) and 2nd in factuality verification (F1 = 0.84), demonstrating the strength of our unified approach.