Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment
nanjing university · Shanghai Jiao Tong University · University of Bristol · Hong Kong Polytechnic University · University of Hong Kong · Carnegie Mellon University
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Most multimodal models treat every negative pair alike, ignoring the ambiguous negatives that differ from the positive by only a small detail. We propose Boundary-A ware Curriculum with Local Attention(BACL), a lightweight add-on that turns these borderline cases into a curriculum signal. A Boundary-aware Negative Sampler gradually raises difficulty, while a Contrastive Local Attention loss highlights where the mismatch occurs. The two modules are fully differentiable and work with any off-the-shelf dual encoder. Theory predicts a fast $\tilde{\mathcal{O}}(1/n)$ error rate; practice shows up to +32 \% R@1 over CLIP and new SOTA on four large-scale benchmarks, all without extra labels.