Establishing Best Practices in Building Rigorous Agentic Benchmarks
UIUC · University of Illinois at Urbana-Champaign · Salesforce Research · Stanford / Berkeley · University of California, Berkeley · Yale University · Princeton University · MIT · MLCommons Association · University of Oxford / Martian · Amazon.com · Amazon Alexa AI · Virginia Polytechnic Institute and State University · University College London, University of London · Imperial College London, Number 10 Downing Street · UK AI Security Institute · UC Berkeley, Transluce AI · Transluce · UC Berkeley · Stanford University
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench-Verified uses insufficient test cases, while $\tau$-bench counts empty responses as successes. Such issues can lead to under- or overestimation of agents’ performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces performance overestimation by 33%.