M3D-VTON: A Monocular-to-3D Virtual Try-On Network

• Published in: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 13219 - 13229
• Main Authors: Zhao, Fuwei, Xie, Zhenyu, Kampffmeyer, Michael, Dong, Haoye, Han, Songfang, Zheng, Tianxiang, Zhang, Tao, Liang, Xiaodan
• Format: Conference Proceeding Journal Article
• Language: English
• Published: IEEE 01.01.2021
• Subjects: 3D from a single image and shape-from-x; Computer vision; Datasets and evaluation; Electronic commerce; Estimation; Fuses; Image and video manipulation detection and integrity methods; Image and video synthesis; Shape; Three-dimensional displays
• ISSN: 1550-5499; 2380-7504; 2380-7504
• DOI: 10.1109/ICCV48922.2021.01299

Virtual 3D try-on can provide an intuitive and realistic view for online shopping and has a huge potential commercial value. However, existing 3D virtual try-on methods mainly rely on annotated 3D human shapes and garment templates, which hinders their applications in practical scenarios. 2D virtual try-on approaches provide a faster alternative to manipulate clothed humans, but lack the rich and realistic 3D representation. In this paper, we propose a novel Monocular-to-3D Virtual Try-On Network (M3D-VTON) that builds on the merits of both 2D and 3D approaches. By integrating 2D information efficiently and learning a mapping that lifts the 2D representation to 3D, we make the first attempt to reconstruct a 3D try-on mesh only taking the target clothing and a person image as inputs. The proposed M3D-VTON includes three modules: 1) The Monocular Prediction Module (MPM) that estimates an initial full-body depth map and accomplishes 2D clothes-person alignment through a novel two-stage warping procedure; 2) The Depth Refinement Module (DRM) that refines the initial body depth to produce more detailed pleat and face characteristics; 3) The Texture Fusion Module (TFM) that fuses the warped clothing with the non-target body part to refine the results. We also construct a high-quality synthesized Monocular-to-3D virtual try-on dataset, in which each person image is associated with a front and a back depth map. Extensive experiments demonstrate that the proposed M3D-VTON can manipulate and reconstruct the 3D human body wearing the given clothing with compelling details and is more efficient than other 3D approaches. 1