Self-Supervised Human Mesh Recovery from Partial Point Cloud via a Self-Improving Loop

摘要

Accurate 3D human mesh recovery from point clouds remains challenging. Most existing methods depend on full 3D supervision or complete input data, both of which are difficult to obtain in practice. %cr update This calls for robust solutions capable of handling partial point clouds in a self-supervised manner. However, the incompleteness of point clouds and the absence of supervision signals pose dual challenges. To tackle these challenges, This calls for robust solutions to handle partial point clouds in a self-supervised manner. To tackle the dual challenges of point cloud incompleteness and the absence of supervision signals, we propose a novel method named SS-HMR, which offers three key insights. First, we estimate point-wise semantics in a self-supervised manner to match partial inputs with a canonical template. The resulting correspondences serve as supervision signals for the regression network in human mesh recovery. Second, we incorporate regression-based and optimization-based paradigms into a self-improving loop: the regression network provides strong initialization for optimization, while the optimization routine generates pseudo-labels that, in turn, enhance the regression network. This mutual feedback enables more accurate and stable mesh recovery over time. Third, generating multiple initializations and selecting the best result mitigates the optimization routine's sensitivity to initialization, improving robustness to sparse and noisy data. %cr update Third, to mitigate sensitivity to initialization in the optimization routine, we generate diverse initialization candidates and transform the challenge of escaping local optima into a controllable selection task, improving robustness against sparse and noisy data. Extensive experiments are conducted on three public datasets and results demonstrate that SS-HMR outperforms existing methods. Notably, SS-HMR performs excellently on different test data, whether from original point clouds captured by depth cameras or LiDAR devices, or from noise-added ones. This shows that SS-HMR has strong generalization ability and robustness across different data sources. Codes are available at https://github.com/suchang-99/SS-HMR.