InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks
Zhejiang University · Rochester Institute of Technology · national university of singaore, National University of Singapore · Bytedance · ByteDance Inc. · Fudan University · ByteDance AI Lab · SalesForce.com · Peking University · Shannon.AI · Alibaba · Zhejiang University, China
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
In this paper, we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks. Agents need to solve these tasks end-to-end by interacting with an execution environment. This benchmark contains DAEval, a dataset consisting of 603 data analysis questions derived from 124 CSV files, and an agent framework which incorporates LLMs to serve as data analysis agents for both serving and evaluating. Since data analysis questions are often open-ended and hard to evaluate without human supervision, we adopt a format-prompting technique to convert each question into a closed-form format so that they can be automatically evaluated. Our extensive benchmarking of 34 LLMs uncovers the current challenges encountered in data analysis tasks. In addition, building upon our agent framework, we develop a specialized agent, DAAgent, which surpasses GPT-3.5 by 3.9% on DABench. Evaluation datasets and toolkits for InfiAgent-DABench are released at https://github.com/InfiAgent/InfiAgent.