Position: AI Evaluations Should be Grounded on a Theory of Capability

摘要

Evaluations of generative models are now ubiquitous, and their outcomes critically shape public and scientific expectations of AI's capabilities. Yet skepticism about their reliability continues to grow. How can we know that a reported accuracy genuinely reflects a model’s underlying performance? Although benchmark results are often presented as direct measurements of capability, in practice they are inferences: treating a score as evidence of capability already presupposes a theory of what it means to be capable at a task. We argue that AI evaluations should instead be framed as inference tasks grounded on an explicit theory of capability. While this perspective is standard in fields like psychometrics, it remains underdeveloped in AI evaluation, where core assumptions are often left *implicit*. As a proof-of-concept, we empirically show that reported performance can depend strongly on the evaluator’s modeling assumptions, underscoring the need for transparent, theory-driven evaluation practices. We conclude by offering practical guidelines for rigorously designing evaluations built on explicit theories of capability.