Generative Multiplane Images: Making a 2D GAN 3D-Aware

• Published in: Computer Vision - ECCV 2022 Vol. 13665; pp. 18 - 35
• Main Authors: Zhao, Xiaoming, Ma, Fangchang, Güera, David, Ren, Zhile, Schwing, Alexander G., Colburn, Alex
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer 2022; Springer Nature Switzerland
• Series: Lecture Notes in Computer Science
• Subjects: 3D-aware generation; GANs; Multiplane images
• ISBN: 9783031200649; 3031200640
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-031-20065-6_2

What is really needed to make an existing 2D GAN 3D-aware? To answer this question, we modify a classical GAN, i.e., StyleGANv2, as little as possible. We find that only two modifications are absolutely necessary: 1) a multiplane image style generator branch which produces a set of alpha maps conditioned on their depth; 2) a pose-conditioned discriminator. We refer to the generated output as a ‘generative multiplane image’ (GMPI) and emphasize that its renderings are not only high-quality but also guaranteed to be view-consistent, which makes GMPIs different from many prior works. Importantly, the number of alpha maps can be dynamically adjusted and can differ between training and inference, alleviating memory concerns and enabling fast training of GMPIs in less than half a day at a resolution of 10242\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1024^2$$\end{document}. Our findings are consistent across three challenging and common high-resolution datasets, including FFHQ, AFHQv2 and MetFaces.