P PaperCompassAI 论文工作台
← 返回论文检索
ICLR 2025PosterAccept (Spotlight)

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya K Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemiński, Gabriel de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan S Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia moakhar, Ayush Tarun, Azmine Toushik Wasi, Thenuka Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut

EPFL · EPFL - EPF Lausanne · Saarland University · Christ University · ServiceNow · University of Toronto · Swiss Federal Institute of Technology Lausanne (EPFL) · Indian Institute of Technology, Bombay · University of California, Santa Barbara · University of the People · Taras Shevchenko National University of Kyiv · Trulioo · Queen Mary University London · African Leadership University · Universidad del Valle del Cauca · ETHZ - ETH Zurich · University of Bath · Bangladesh University of Engineering and Technology · Islamic Azad University Science and Research Branch · Beijing Academy of Artificial Intelligence (BAAI) · University of Waterloo · Universität Mannheim · Mohamed bin Zayed University of Artificial Intelligence · University of Cambridge · Instituto Tecnológico de Aeronáutica · LIMSI - CNRS · Cohere For AI · Harvey · Winterlight Labs · University of Montreal/Mila · Aalborg University · Tribhuvan University · Emory University · Meta · KTH Royal Institute of Technology · Adobe Systems · Cohere for AI Community · University of New York Tirana · Sharif · Sharif University of Technology · Shahjalal University of Science and Technology · University of Colombo School of Computing · Sabanci University · Aalborg University (Copenhagen)

论文页面 ↗ 登录后记笔记

PDF 由论文原始站点提供，PaperCompass 不保存论文文件。

摘要

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts.Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.