Distilling and Scaling Hierarchical Vision Transformer to 30B Parameters

摘要

Recent efforts have successfully scaled plain Vision Transformers (ViTs) to unprecedented sizes, ranging from 6B to 22B parameters. However, hierarchical ViTs, which are inherently better suited for multi-scale representation and dense prediction tasks, have largely remained constrained to under 2B parameters. This scaling gap prevents the community from leveraging high-capacity, multi-scale foundation models for complex visual scenes. To bridge this gap, we propose EHV, an Efficient Hierarchical ViT architecture designed for massive scaling. Our dense models scale from 200M to 5B parameters, and we further introduce a Sparse Mixture-of-Experts (SMoE) variant, pushing the scale to an industry-leading 30B parameters. The training pipeline follows a two-stage pretraining strategy: (1) MAE Stage: self-supervised pretraining on ImageNet-21K using a Masked Autoencoder (MAE); and (2) Distillation Stage: distilling knowledge from multiple state-of-the-art foundation models on a 27M-image dataset. With only 6.7B active parameters, EHV-5B-MoE achieves an exceptional 89.0% linear accuracy on ImageNet-1K, outperforming much larger models like EVA-CLIP-18B and DINOv3-7B. These results demonstrate that EHV effectively learns high-quality, generalizable, and linearly separable features across image classification, video analysis, and dense prediction tasks.