SWINGARENA: Adversarial Programming Arena for Long-context GitHub Issue Solving
HKU UniPat AI · University of Hong Kong · UCLA Computer Science Department, University of California, Los Angeles · Southern University of Science and Technology · Tsinghua University · University of Michigan - Ann Arbor · Ohio State University, Columbus · Huawei Technologies Ltd. · The University of Hong Kong · University of Michigan · Stanford University · HKUST (GZ) · Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Computer Science, The University of Hong Kong · University of California, Los Angeles
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
We present \textsc{SwingArena}, a adversarial evaluation framework for Large Language Models (LLMs) that closely mirrors real-world software development workflows. Unlike traditional static benchmarks, \textsc{SwingArena} models the collaborative process of software iteration by pairing LLMs as \textit{submitters}, who generate patches, and \textit{reviewers}, who create test cases and verify the patches through continuous integration (CI) pipelines. To support these interactive evaluations, we introduce a retrieval-augmented code generation (RACG) module that efficiently handles long-context challenges by providing syntactically and semantically relevant code snippets from large codebases, supporting multiple programming languages (C++, Python, Rust, and Go). This enables the framework to scale across diverse tasks and contexts while respecting token limitations. Our experiments, using over 400 high-quality real-world GitHub issues selected from a pool of 2,300 issues, show that models like GPT-4o excel at aggressive patch generation, whereas DeepSeek and Gemini prioritize correctness in CI validation. \textsc{SwingArena} presents a scalable and extensible methodology for evaluating LLMs in realistic, CI-driven software development settings.