Let’s Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models’ Understanding of Sports
Indian Institute of Technology, Patna · Bharatiya Vidya Bhavan’s Sardar Patel Institute of Technology · Universitas Gadjah Mada · King Mongkut’s Institute of Technology Ladkrabang · Shenzhen Technology University · Université Paul Sabatier / Université de Toulouse III
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。DOI 10.18653/v1/2025.emnlp-main.769 ↗
摘要
Language Models (LMs) are primarily evaluated on globally popular sports, often overlooking regional and indigenous sporting traditions. To address this gap, we introduce CultSportQA, a benchmark designed to assess LMs’ understanding of traditional sports across 60 countries and 6 continents, encompassing four distinct cultural categories. The dataset features 33,000 multiple-choice questions (MCQs) across text and image modalities, categorized into primarily three key types: history-based, rule-based, and scenario-based. To evaluate model performance, we employ zero-shot, few-shot, and chain-of-thought (CoT) prompting across a diverse set of Large Language Models (LLMs), Small Language Models (SLMs), and Multimodal Large Language Models (MLMs). By providing a comprehensive multilingual and multicultural sports benchmark, CultSportQA establishes a new standard for assessing AI’s ability to understand and reason about traditional sports. The dataset will be publicly available, fostering research in culturally aware AI systems.