MMTEB: Massive Multilingual Text Embedding Benchmark
Aarhus University · Zendesk · Esker - INSA Lyon, LIRIS · Independent Researcher · Mohamed bin Zayed University of Artificial Intelligence · Komodo Health · University of Cambridge · Capital One · Jina AI · Micorsoft Research · Wikit · Institut National des Sciences Appliquées de Rouen · Max-Planck-Institute for Intelligent Systems, Max-Planck Institute · icodeformybhasa · University of Oxford · ITMO University · Koç University · Apart Research, Humane Intelligence · Beijing Academy of Artificial Intelligence · Weav.AI · Microsoft · National Information Processing Institute · DataChat · ellamind GmbH · Universität Hamburg · University of Wuppertal · NVIDIA · Peking University & MSRA · CentraleSupelec · Hugging Face · Wrocław University of Science and Technology · Korea University HIAI Research · Comenius University in Bratislava · University of Waterloo · Independent · University of Zurich · Princeton University · Federal Research Center «Computer Science and Control» of Russian Academy of Sciences · SAP · DocuSign · Microsoft Research · University of Roma "La Sapienza" · University of Pennsylvania, University of Pennsylvania · SaluteDevices · University of Amsterdam · University of Washington, Seattle · McGill University · Chattermill · R V College of Engineering · Higher School of Economics, Sber Devices · Robert Koch Institute · Sber · Nirma University · Deutsche Telekom · Research, Microsoft · Allen Institute for Artificial Intelligence · SaluteDevices, HSE University · Tano Labs · The London Institute of Banking & Finance · Cornell University · Northeastern University · Cohere For AI · Durham University · Johns Hopkins University · McGill University / Mila · Stanford University
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Text embeddings are typically evaluated on a narrow set of tasks, limited in terms of languages, domains, and task types. To circumvent this limitation and to provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) -- a large-scale community-driven initiative expanding MTEB to over 500 quality-controlled evaluation tasks across 1,000+ languages. MMTEB includes a wide range of challenging novel tasks such as instruction following, long-document retrieval, and code retrieval, and represents the largest multilingual collection of evaluation tasks for embedding models to date. We use this collection to construct multiple highly multilingual benchmarks. We evaluate a representative set of models on these benchmarks.Our findings indicate that, while LLM-based models can achieve state-of-the-art performance on a subset of languages, the best-performing publicly available model across languages is the notably smaller, multilingual-e5-large-instruct.Massive benchmarks often impose high computational demands, limiting accessibility, particularly for low-resource communities. To address this, we downsample tasks based on inter-task correlation (i.e., selecting only a diverse set of tasks) while preserving relative rankings.We further optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks at a significantly lower computational cost. For instance, we introduce a new zero-shot English benchmark that maintains a similar ordering at a fraction of the cost.