Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

摘要

Foundation models have driven rapid progress in computer vision, yet the two dominant paradigm, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose **GPUA**, a *Geometry-Preserving Unsupervised Alignment* framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a *visual language* and learns an *orthogonal* mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap *without labels, backpropagation, or model parameter updates*. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead.