RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback
University of Illinois Chicago · Tsinghua University · Cornell University · The Chinese University of Hong Kong · University of Illinois at Chicago · Mohamed bin Zayed University of Artificial Intelligence · Chinese University of Hong Kong · Tongji University · ServiceNow Inc · Shanghai Jiao Tong University · National Taiwan University · Xi'an Jiaotong University · Department of Computer Science, University of Illinois Chicago · Department of Computer Science and Engineering, The Chinese University of Hong Kong · University of illinois Chicago · University of Illinois of Chicago · MBZUAI · University of Illinois, Chicago
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Large language models (LLMs) show the promise in supporting scientific research implementation, yet their ability to generate correct and executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic workflows of scientific research development. To address this gap, we present RECODE-H, a benchmark of 102 tasks from research papers and repositories that evaluates LLMs through multi-turn interactions with human feedback. It includes structured instructions, unit tests, and a five-level feedback hierarchy to reflect realistic researcher–agent collaboration. We further present ReCodeAgent, a framework that integrates feedback into iterative code generation. Experimentswith leading LLMs, including GPT-5, Claude-Sonnet-4, DeepSeek-V3.1, and Gemini 2.5, show substantial performance gains with richer feedback, while also highlighting ongoing challenges in the generation of complex research code. RECODE-H establishes a foundation for developing adaptive, feedback-driven LLM agents in scientific research implementation.