Many-shot Jailbreaking
University of Toronto · Anthropic · University of Oxford · CZ Biohub · Stanford University · Anthropic PBC · Funded by ARC Theory · Harvard University · Constellation · Yale University · Machine Intelligence Research Institute · Anthropic/Harvard · Massachusetts Institute of Technology · UK AISI · NYU + Anthropic
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This attack is newly feasible with the larger context windows recently deployed by language model providers like Google DeepMind, OpenAI and Anthropic. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.