Boosting Visual Reprogramming for CLIP with Dual Granularity Alignment

摘要

Model reprogramming adapts pretrained models to downstream tasks by modifying their input and output spaces. Visual reprogramming, as a prominent instance, has been explored in pioneer works on CLIP, which introduces learnable input transformations as visual prompts to repurpose its visual-language alignment for downstream visual tasks. Existing VR methods focus on single-level alignment between prompted images and text descriptions, overlooking inherent structural information in data that facilitates alignment: semantic granularity from label hierarchies and visual granularity from multi-scale representations. To address this gap, we propose Dual Granularity Alignment (DGA) with two key components for multi-level fusion. For visual granularity, we generate multi-scale images and introduce Uncertainty-calibrated Prediction Fusion (UPF), which fuses predictions based on uncertainty estimation to capture hierarchical spatial information. For semantic granularity, we construct category hierarchies via Prototype-guided Label Hierarchization and develop Hierarchical Knowledge Propagation (HKP), which transfers superclass knowledge for coherent multi-level visual prompts alignment. Our DGA collaboratively integrates both granularities to enhance alignment effectiveness. Experiments across 12 downstream datasets demonstrate DGA's superiority over baselines on both ViT-based and ResNet-based CLIP architectures. Specifically, DGA achieves a 4.5% improvement over the previous state-of-the-art method on ViT-16-based CLIP. By explicitly modeling structural granularities, DGA establishes a new paradigm for visual reprogramming.