Measuring what Matters: Construct Validity in Large Language Model Benchmarks
University of Oxford · EPFL · Technical University Munich (TUM) · École polytechnique fédérale de Lausanne (EPFL) · Centre for Digital Governance, Hertie School · University of Oxford, DeepMind Scholar · Stanford University · FAIR at Meta, University of Oxford · Imperial College London · Universidad Politécnica de Madrid / SomosNLP · Stanford University / LegalAI · Allen Institute for AI · University of Oxford / UK AISI · University of Oxford Google DeepMind · Department of Computer Science, Yale University · AI Security Institute · EPFL - EPF Lausanne · Yale University · Meedan · UC Berkeley · UK AI Security Institute
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as `safety' and `robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.