SciCode: A Research Coding Benchmark Curated by Scientists
UIUC · Mistral AI · University of Illinois · University of Illinois at Urbana-Champaign · University of North Carolina at Chapel Hill · University of California, Los Angeles · University of Chicago · University of Texas at Austin · Princeton University · University of Illinois Urbana-Champaign · University of Washington · Carnegie Mellon University · Argonne National Laboratory and The University of Chicago · Department of Computer Science, University of Illinois Urbana-Champaign
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Since language models (LMs) now outperform average humans on many challenging tasks, it is becoming increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this by examining LM capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we create a scientist-curated coding benchmark, SciCode. The problems naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems, and it offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. OpenAI o1-preview, the best-performing model among those tested, can solve only 7.7\% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards realizing helpful scientific assistants and sheds light on the building and evaluation of scientific AI in the future.