EgoHierMask: Hierarchical Semantic-Prior Guided Masked Autoencoder for Egocentric Action Recognition

摘要

Egocentric action recognition holds critical value in augmented reality, embodied AI, and human behavior analysis. While transformer-based masked autoencoders show potential in general video representation learning, their direct application to egocentric vision faces fundamental limitations --- random masking strategies disrupt crucial spatiotemporal features like hand-object interaction hierarchies and viewpoint dynamics by neglecting task-specific semantic priors. Through systematic analysis, this paper reveals three complementary semantic priors for egocentric video understanding: verb-centric motion patterns characterizing hand trajectories, noun-aware attention regions highlighting object contact points, and action-oriented global context integrating holistic semantics. These hierarchical cues address egocentric visual specificity through motion granularity, interaction locality, and semantic integrity. Building on this discovery, we propose EgoHierMask: a hierarchical semantic prior-guided masked autoencoder framework coordinating vision-language knowledge through differentiated masking strategies. The framework employs frozen vision-language teacher models to generate multi-level semantic attention maps, systematically guiding three specialized masking branches: a) dynamic motion masking preserves hand movement continuity through temporal verb attention, b) interaction-sensitive masking maintains object manipulation coherence via spatial noun saliency, and c) spatiotemporal joint masking encodes complete action semantics through global context alignment. Additionally, to enhance learning efficacy, we curate a distribution-balanced pretraining corpus and devise a unified architecture with dual-granularity supervision, combining pixel-level reconstruction with semantic-level distillation within the VideoMAE paradigm.Extensive experiments demonstrate state-of-the-art performance across major benchmarks, validating the crucial value of hierarchical prior injection for egocentric representation learning.