MRBench: A Multi-Image Reasoning Benchmark with Adaptive Knowledge Retrieval

摘要

Multi-image understanding is crucial in real-world applications such as social media analysis and news reporting. However, existing benchmarks fall short in evaluating models' ability to integrate external knowledge and perform cross-image reasoning. To address this gap, we introduce MRBench, a comprehensive benchmark designed to assess knowledge-based reasoning across 12 diverse domains, incorporating four types of image relations: visually similar, identical entities, attribute-associated, and independent images. Additionally, we propose Multimodal Adaptive Retrieval Reasoning (MARR), a novel framework that enables the analysis of relationships among multiple input images and adaptively determines when to terminate the retrieval process. Extensive evaluations of state-of-the-art multimodal large language models (MLLMs) show a notable gap between model and human performance. The best-performing model, Gemini 2.0, reaches 56.86% accuracy, still 20.24% below humans. Proprietary models generally surpass open-source ones, particularly on visually similar and same-entity tasks, underscoring current limits in multi-image reasoning and retrieval and positioning MRBench as a key diagnostic tool. Our benchmark is available for further research and development in this field. https://github.com/Bruce-XJChen/MRBench.