FinBen: A Holistic Financial Benchmark for Large Language Models
Yale University · Wuhan University · New York University · SUN YAT-SEN UNIVERSITY · Columbia University · Sichuan University · University of California, San Diego · Jiangxi Normal University · The University of Manchester · University of Manchester · School of Engineering and Applied Sciences, Harvard University · Stevens Institute of Technology · Seeking for Ph.D. Positions · Yunnan University · NanJing Audit University · Columbia University & Rensselaer Polytechnic Institute · University of Florida · Universita' degli studi di Padova · Southwest Jiaotong University · The Fin AI
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
LLMs have transformed NLP and shown promise in various fields, yet their potential in finance is underexplored due to a lack of comprehensive benchmarks, the rapid development of LLMs, and the complexity of financial tasks. In this paper, we introduce FinBen, the first extensive open-source evaluation benchmark, including 42 datasets spanning 24 financial tasks, covering eight critical aspects: information extraction (IE), textual analysis, question answering (QA), text generation, risk management, forecasting, decision-making, and bilingual (English and Spanish). FinBen offers several key innovations: a broader range of tasks and datasets, the first evaluation of stock trading, novel agent and Retrieval-Augmented Generation (RAG) evaluation, and two novel datasets for regulations and stock trading. Our evaluation of 21 representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals several key findings: While LLMs excel in IE and textual analysis, they struggle with advanced reasoning and complex tasks like text generation and forecasting. GPT-4 excels in IE and stock trading, while Gemini is better at text generation and forecasting. Instruction-tuned LLMs improve textual analysis but offer limited benefits for complex tasks such as QA. FinBen has been used to host the first financial LLMs shared task at the FinNLP-AgentScen workshop during IJCAI-2024, attracting 12 teams. Their novel solutions outperformed GPT-4, showcasing FinBen's potential to drive innovations in financial LLMs. All datasets and code are publicly available for the research community, with results shared and updated regularly on the Open Financial LLM Leaderboard.