Design, Implementation, and Evaluation of a Point Cloud Codec for Tele-Immersive Video

• Published in: IEEE transactions on circuits and systems for video technology Vol. 27; no. 4; pp. 828 - 842
• Main Authors: Mekuria, Rufael, Blom, Kees, Cesar, Pablo
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.04.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Cloud computing; Codec; Codecs; Color; Data compression; Encoding; Image coding; Image compression; Image reconstruction; Iterative algorithms; JPEG encoders-decoders; Mapping; Mixed reality; MPEG encoders; Octrees; point clouds; Predictions; Quaternions; Real time; Real-time systems; Rendering (computer graphics); State of the art; Streaming media; Submerging; teleconferencing; Three dimensional models; Three-dimensional displays; video codecs; Video compression; virtual reality
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2016.2543039

We present a generic and real-time time-varying point cloud codec for 3D immersive video. This codec is suitable for mixed reality applications in which 3D point clouds are acquired at a fast rate. In this codec, intra frames are coded progressively in an octree subdivision. To further exploit interframe dependencies, we present an inter-prediction algorithm that partitions the octree voxel space in N × N × N macroblocks (N = 8, 16, 32). The algorithm codes points in these blocks in the predictive frame as a rigid transform applied to the points in the intra-coded frame. The rigid transform is computed using the iterative closest point algorithm and compactly represented in a quaternion quantization scheme. To encode the color attributes, we defined a mapping of color per vertex attributes in the traversed octree to an image grid and use legacy image coding method based on JPEG. As a result, a generic compression framework suitable for realtime 3D tele-immersion is developed. This framework has been optimized to run in real time on commodity hardware for both the encoder and decoder. Objective evaluation shows that a higher rate-distortion performance is achieved compared with available point cloud codecs. A subjective study in a state-of-the-art mixed reality system shows that introduced prediction distortions are negligible compared with the original reconstructed point clouds. In addition, it shows the benefit of reconstructed point cloud video as a representation in the 3D virtual world. The codec is available as open source for integration in immersive and augmented communication applications and serves as a base reference software platform in JTC1/SC29/WG11 (MPEG) for the further development of standardized point-cloud compression solutions.