IEEE Transactions on Image Processing (TIP)

Pose2UV: Single-shot Multi-person Mesh Recovery with Deep UV Prior

     

Given a challenging multi-person image, we first utilize the predicted 2D pose to locate and crop each individual. The heatmaps and image patch of each person are then fed to the visible pose-mask module (a) to estimate the visible heatmaps and masks. With the help of the proposed UVPrior (c), the UV Prediction Module (b) regresses a plausible UV position map from partial body cues. We can resample the predicted map to obtain a human mesh. Finally, we calculate the absolute position based on the regressed 3D joints and the visible 2D joints (d).

Abstract

In this work, we focus on the task of multi-person mesh recovery from a single color image, where the key issue is to tackle the pixel-level ambiguities caused by inter-person occlusions. Overall, there are two main technical challenges when addressing the ambiguities: how to extract valid target features under occlusions and how to reconstruct reasonable human meshes with only a handful of body cues? To deal with these problems, our key idea is to utilize the predicted 2D poses to locate and separate the target person, and reconstruct them with a novel learning-based UV prior. Specifically, we propose a visible pose-mask module to help extract valid target features, then train a dense body mesh prior to promote reconstructing natural mesh represented by the UV position map. To evaluate the performance of our proposed method under occlusions, we further build an in-the-wild 3D multi-person benchmark named as 3DMPB. Experimental results demonstrate that our method achieves state-of-the-art compared with previous methods.

3DMPB Dataset

We build an in-the-wild 3D multi-person benchmark, , which provides a lot of human-human interactions and inter-person occlusions cases with challenging poses in real basketball scenes. The annotations are obtained with DMMR.

The rights to copy, distribute, and use the 3DMPB dataset (henceforth called "3DMPB") you are being given access to are under the control of Yangang Wang, director of the Vision and Cognition Lab, Southeast University. You are hereby given permission to copy this data in electronic or hardcopy form for your own scientific use and to distribute it for scientific use to colleagues within your research group. Inclusion of images or video made from this data in a scholarly publication (printed or electronic) is also permitted. In this case, credit must be given to the publication: *Pose2UV: Single-shot Multi-person Mesh Recovery with Deep UV Prior*. For any other use, including distribution outside your research group, written permission is required from Yangang Wang. . Commercial use includes but is not limited to sale of the data, derivatives, replicas, images, or video, inclusion in a product for sale, or inclusion in advertisements (printed or electronic), on commercially-oriented web sites, or in trade shows.

Results

Materials

Reference

Acknowledgments: This work was supported in part by the National Natural Science Foundation of China (No. 62076061), the “Young Elite Scientists Sponsorship Program by CAST” (No. YES20200025), and the “Zhishan Young Scholar” Program of Southeast University (No. 2242021R41083).