Sequential Point Cloud Upsampling by Exploiting Multi-Scale Temporal Dependency

• Published in: IEEE transactions on circuits and systems for video technology Vol. 31; no. 12; pp. 4686 - 4696
• Main Authors: Wang, Kaisiyuan, Sheng, Lu, Gu, Shuhang, Xu, Dong
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: dynamic point cloud sequences; Estimation; Feature extraction; Point cloud compression; Point cloud upsampling; Solid modeling; spatio-temporal exploration; Spatiotemporal phenomena; Superresolution; Three-dimensional displays
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2021.3104304

In this work, we propose a new sequential point cloud upsampling method called SPU, which aims to upsample sparse, non-uniform, and orderless point cloud sequences by effectively exploiting rich and complementary temporal dependency from multiple inputs. Specifically, these inputs include a set of multi-scale short-term features from the 3D points in three consecutive frames (i.e., the previous/current/subsequent frame) and a long-term latent representation accumulated throughout the point cloud sequence. Considering that these temporal clues are not well aligned in the coordinate space, we propose a new temporal alignment module (TAM) based on the cross-attention mechanism to transform each individual feature into the feature space of the current frame. We also propose a new gating mechanism to learn the optimal weights for these transformed features, based on which the transformed features can be effectively aggregated as the final fused feature. The fused feature can be readily fed into the existing single frame-based point cloud upsampling methods (e.g., PU-Net, MPU and PU-GAN) to generate the dense point cloud for the current frame. Comprehensive experiments on three benchmark datasets DYNA, COMA, and MSR Action3D demonstrate the effectiveness of our method for upsampling point cloud sequences.