Neural foveated super‐resolution for real‐time VR rendering

• Published in: Computer animation and virtual worlds Vol. 35; no. 4
• Main Authors: Ye, Jiannan, Meng, Xiaoxu, Guo, Daiyun, Shang, Cheng, Mao, Haotian, Yang, Xubo
• Format: Journal Article
• Language: English
• Published: Chichester Wiley Subscription Services, Inc 01.07.2024
• Subjects: Algorithms; foveated rendering; Image quality; Image reconstruction; Neural networks; Rendering; super‐resolution; Virtual reality; Visual acuity
• ISSN: 1546-4261; 1546-427X
• DOI: 10.1002/cav.2287

As virtual reality display technologies advance, resolutions and refresh rates continue to approach human perceptual limits, presenting a challenge for real‐time rendering algorithms. Neural super‐resolution is promising in reducing the computation cost and boosting the visual experience by scaling up low‐resolution renderings. However, the added workload of running neural networks cannot be neglected. In this article, we try to alleviate the burden by exploiting the foveated nature of the human visual system, in a way that we upscale the coarse input in a heterogeneous manner instead of uniform super‐resolution according to the visual acuity decreasing rapidly from the focal point to the periphery. With the help of dynamic and geometric information (i.e., pixel‐wise motion vectors, depth, and camera transformation) available inherently in the real‐time rendering content, we propose a neural accumulator to effectively aggregate the amortizedly rendered low‐resolution visual information from frame to frame recurrently. By leveraging a partition‐assemble scheme, we use a neural super‐resolution module to upsample the low‐resolution image tiles to different qualities according to their perceptual importance and reconstruct the final output adaptively. Perceptually high‐fidelity foveated high‐resolution frames are generated in real‐time, surpassing the quality of other foveated super‐resolution methods. We propose a method to improve neural super‐resolution for realtime rendering and alleviate the computation burden in VR by exploiting the foveated nature of the human visual system. By leveraging a partition‐assemble scheme, our method adaptively upsamples the low‐resolution image tiles to different qualities according to their perceptual importance.