AetherCode: Evaluating LLMs’ Ability to Win In Premier Programming Competitions
Peking Univerdity · Bytedance · ByteDance Inc. · Department of Computer Science and Engineering, Hong Kong University of Science and Technology · Columbia University · Singapore University of Technology and Design · Fudan University · WITS Corp · Beijing University of Aeronautics and Astronautics · University of the Chinese Academy of Sciences · Beijing Normal University · , Chinese Academy of Sciences
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present **AetherCode**, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning.