MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Shanghai Artificial Intelligent Laboratory · Shanghai Jiaotong University · The University of Hong Kong · University of Hong Kong · shanghai jiaotong university · University of Adelaide · Zhejiang university · Shanghai AI lab · MMLab of CUHK & Shanghai AI Laboratory · Zhejiang University · shanghai ai lab · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Shanghai Aritifcal Intelligence Laboratory · Shanghai AI Laboratory · Shanghai Artificial Intelligence Innovation Center
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Large Vision-Language Models (LVLMs) show significant strides in general-propose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, and reasoning. MMT-Bench comprises $31,325$ meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering $32$ core meta-tasks and $162$ subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving $20$ publicly available LVLMs such as the proprietary GeminiProVision model, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.