Reflective RAG: Self-Evaluation Driven Strategy Optimization in Agentic Retrieval-Augmented Generation

摘要

Retrieval-Augmented Generation (RAG) has emerged as a widely adopted paradigm for grounding Large Language Models (LLMs) in external knowledge. Recent agentic RAG systems introduce multi-turn reasoning, but they often lack the capacity to evaluate the utility of retrieved information, leading to brittle reasoning and suboptimal decision-making. We propose Reflective RAG, an agentic framework that incorporates self-evaluation to dynamically optimize retrieval and generation strategy. At its core, Reflective RAG employs a reflection tagging mechanism that allows the model to critique the relevance of retrieved content, thereby explicitly guiding its subsequent policy. To ensure robust learning, we introduce a two-stage training procedure that partially decouples evaluation semantics from strategy optimization. First, during supervised fine-tuning (SFT), the model learns to generate accurate reflection signals by self-correcting labels based on internal uncertainty. Second, a reinforcement learning (RL) stage optimizes the agent’s strategy using these reflections, stabilized by dynamic KL regularization. Evaluations across five knowledge-intensive QA benchmarks demonstrate that Reflective RAG consistently outperforms strong agentic baselines. Further analysis demonstrates its improved training stability and stronger generalization to complex multi-hop reasoning tasks.