Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation
Copenhagen University · LIAA - Institute of Computer Sciences, CONICET & Universidad de Buenos Aires · Korea Advanced Institute of Science & Technology (KAIST) · Sharif · Christ University · ETHZ - ETH Zurich · EPFL · Taras Shevchenko National University of Kyiv · Amazon · University of Copenhagen · University of Cambridge · Vanguard · Stanford · University of Bath · ServiceNow · University of California, Santa Barbara · Emory University · Tribhuvan University · Uppsala University · Karlsruher Institut für Technologie · Comenius University in Bratislava · Shahjalal University of Science and Technology · Aivar · T-Tech · Higher School of Economics · Sharif University of Technology · Universidade de São Paulo · Islamic Azad University Science and Research Branch · Iran University of Science and Technology Tehran, University of Tehran · AI Circle · Mohamed bin Zayed University of Artificial Intelligence · Cohere for AI Community · Universidade Federal de São Carlos · University of Montreal/Mila · Lviv Polytechnic National University · Cohere AI · Universidad de Buenos Aires / CONICET · Cohere For AI
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and language, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.