P PaperCompassAI 论文工作台
← 返回论文检索
ICLR 2025PosterAccept (Poster)

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Javier Rando, Tony Wang, Stewart Slocum, Dmitrii Krasheninnikov, Usman Anwar, Micah Carroll, Xander Davies, Claudia Shi, Thomas Gilbert, Rachel Freedman, Charbel-Raphael Segerie, Phillip Christoffersen, Jacob Pfau, Tomek Korbak, Xin Chen, Lauro Langosco, Samuel Marks, Erdem Bıyık, Dorsa Sadigh, David Krueger, Pedro Freire, Mehul Damani, Jérémy Scheurer, David Lindner, Anca Dragan, Anand Siththaranjan, Dylan Hadfield-Menell, Max Nadeau, Stephen Casper, Peter Hase, Andi Peng, Eric Michaud

ETH Zurich · Massachusetts Institute of Technology · University of Amsterdam · University Of Cambridge, UK · University of California Berkeley · University of Oxford · Columbia University · Cornell Tech · Ecole Normale Superieure · New York University · ETHZ - ETH Zurich · University of Cambridge · Northeastern University · University of Southern California · Stanford University · Ecole Polytechnique · UC Berkeley · University of California, Berkeley · Harvard University · UNC Chapel Hill · MIT

论文页面 ↗ 登录后记笔记

PDF 由论文原始站点提供，PaperCompass 不保存论文文件。

摘要

Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of a multi-layered approach to the development of safer AI systems.