RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation
Amazon · Shanghai Jiaotong University · University of Waterloo · AWS AI Lab · Department of Computer Science and Engineering, Hong Kong University of Science and Technology · Westlake University · Tsinghua University, Tsinghua University · Fudan University · University of Utah · Amazon Web Services · , Institute of automation, Chinese academy of science · Carnegie Mellon University · Shanghai New York Univeristy
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Despite Retrieval-Augmented Generation (RAG) has shown promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems.