Skeleton Compression and Complementary Enhanced Fusion Under Branch-Stage Supervision for Human Action Recognition

摘要

Skeleton-based human action recognition (HAR) is greatly affected by abnormal situations in real-world scenarios, like occlusions and performance limitations of motion capture devices. Although recent research has enhanced the robustness of recognition by incorporating occlusion simulation in model training, it is still insufficient to effectively handle the complex and diverse abnormal situations in real-world scenarios. To address this issue, we propose SCCEAP, a novel framework combining fine-grained Skeleton Compression and Complementary Enhanced Adaptive feature fusion with the supervision of branch-stage text Prompts, for robust skeleton-based HAR. Our contributions lie in three aspects. First, the fine-grained skeleton compression is designed to generate multi-granularity skeleton sequences with diverse spatial details by fusing joints in the human skeleton according to their joint reliabilities and correlations. Then, we devise the complementary enhanced adaptive feature fusion, which utilizes motion details and stable semantic descriptions of motion features of the uncompressed and compressed skeleton sequences respectively, for complementary enhancement and adaptive feature fusion. Third, the branch-stage composite text-prompt supervision is performed to integrate both branch-wise and stage-wise text-prompt supervision for improving the ability to learn fine-grained spatiotemporal relationships of motion features. Experiments on three benchmark datasets-NTU RGB+D, NTU RGB+D 120, and Kinetics-400-demonstrate that SCCEAP achieves the state-of-the-art (SOTA) results, excelling on both normal and noisy skeleton data.