MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
Beijing University of Aeronautics and Astronautics · Institute of automation, Chinese Academy of Sciences; University of Chinese Academy of Sciences · Kuaishou- 快手科技 · stepfun · University of Michigan - Ann Arbor · Alibaba Group · nanjing university · Applied Research Center, Tencent PCG · Fudan University · National University of Singapore · Kuaishou Technology · Key Laboratory of Machine Perception · Chinese Academy of Sciences, China · Nanjing University
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
The advent of Multimodal Large Language Models (MLLMs) has expanded AI capabilities to visual modalities, yet existing evaluation benchmarks remain limited to single-video understanding, overlooking the critical need for multi-video understanding in real-world scenarios (e.g., sports analytics and autonomous driving). To address this significant gap, we introduce **MVU-Eval**, the first comprehensive benchmark for evaluating **M**ulti-**V**ideo **U**nderstanding for MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies through 1,824 meticulously curated question-answer pairs spanning 4,959 videos from diverse domains, addressing both fundamental perception tasks and high-order reasoning tasks. These capabilities are rigorously aligned with real-world applications such as multi-sensor synthesis in autonomous systems and cross-angle sports analytics. Through extensive evaluation of state-of-the-art open-source and closed-source models, we reveal significant performance discrepancies and limitations in current MLLMs' ability to perform understanding across multiple videos.The benchmark will be made publicly available to foster future research.