Closing the Safety Gap: Surgical Concept Erasure in Visual Autoregressive Models
Harbin Institute of Technology, shenzhen · SIGS, Tsinghua University · Jilin University · Harbin Institute of Technology · University of Hong Kong · Harbin Institute of Technology, Shenzhen · Graduate School at Shenzhen, Tsinghua University · Harbin Institute of Technology,Shenzhen · Tsinghua University, Tsinghua University
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
The rapid progress of visual autoregressive (VAR) models has brought new opportunities for text-to-image generation, but also heightened safety concerns. Existing concept erasure techniques, primarily designed for diffusion models, fail to generalize to VARs due to their next-scale token prediction paradigm. In this paper, we first propose a novel VAR Erasure framework **VARE** that enables stable concept erasure in VAR models by leveraging auxiliary visual tokens to reduce fine-tuning intensity. Building upon this, we introduce **S-VARE**, a novel and effective concept erasure method designed for VAR, which incorporates a filtered cross entropy loss to precisely identify and minimally adjust unsafe visual tokens, along with a preservation loss to maintain semantic fidelity, addressing the issues such as language drift and reduced diversity introduce by na\"ive fine-tuning. Extensive experiments demonstrate that our approach achieves surgical concept erasure while preserving generation quality, thereby closing the safety gap in autoregressive text-to-image generation by earlier methods.