Zero Matrix guided Adaptive Image Vaccine against Diffusion Model-based Multitask

摘要

While the widespread adoption of diffusion models in image generation has showcased remarkable capabilities, it has also inadvertently opened the door to malicious exploitation. Recent research has primarily concentrated on protecting images from the misuse of diffusion-based customized generation (CG). However, these approaches often overlook that image details can still be enhanced through diffusion-based super-resolution (SR) techniques, significantly increasing the risks of personal image leakage and abuse. To combat these multifaceted risks, we propose the Zero Matrix-guided Adaptive Image Vaccine (ZMAIV) framework. Specifically, we introduce the Self-attention Removal strategy, tailored for CG, which disrupts the model's core mechanism of focusing on sensitive spaces. Concurrently, the High-frequency Removal strategy is proposed to impede the high-frequency details reconstruction of SR. These defense strategies effectively dismantle the underlying mechanisms that facilitate unauthorized data extrapolation. Moreover, the proposed Adaptive Space Search Attack precisely targets critical spaces within images for vaccine injection, optimizing perturbation placement to minimize perturbation conflict while maintaining defense performance. Extensive experiments demonstrate that the proposed ZMAIV outperforms the state-of-the-arts in the aspects of simultaneously defending against diffusion-based CG and SR, affirming its superiority in safeguarding visual content against these dual threats.