Reinforcement Learning with Evolving Rubrics for Deep Research
University of Washington · CMU, Carnegie Mellon University · Massachusetts Institute of Technology · Cornell University · Department of Computer Science, University of Washington · Ai2 · University of California, Berkeley · Allen Institute for Artificial Intelligence · Microsoft AI · Allen Institute for AI · Meta FAIR · Carnegie Mellon University · MIT
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Deep research agents perform multi-step research to produce long-form, well-attributed answers. However, most open deep research agents are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards, which does not extend to realistic long-form tasks. We address this with **Reinforcement Learning with Evolving Rubrics (RLER)**, where rubrics are constructed and maintained to *co-evolve* with the policy model during training. This allows the rubrics to incorporate newly explored information from search and contrasting model responses, enabling better fact checking and more discriminative on-policy feedback. Using RLER, we develop **Deep Research Tulu (DR Tulu-8B)**, the first fully open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare, and general domains, DR Tulu-8B substantially outperforms existing open deep research agents (by 15.6% over Tongyi DR on average) and matches or exceeds proprietary deep research agents (by 0.7% over OpenAI DR on average), while being significantly smaller and cheaper per query (1000x cheaper than OpenAI DR per query).