Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality

• Published in: IEEE journal on emerging and selected topics in circuits and systems Vol. 9; no. 1; pp. 149 - 162
• Main Authors: Park, Jounsup, Chou, Philip A., Hwang, Jenq-Neng
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.03.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Augmented reality; Bandwidth; Bit rate; Buffers; client buffer management; Computer simulation; Culling; DASH; Digital media; free-viewpoint video; Greedy algorithms; Hologram; Holograms; Holography; Indexes; Media; Optimization; Queues; Streaming media; Three-dimensional displays; Throughput; tile; Tiles; Video transmission; Virtual reality; voxelized point cloud
• ISSN: 2156-3357; 2156-3365
• DOI: 10.1109/JETCAS.2019.2898622

Volumetric media, popularly known as holograms, need to be delivered to users using both on-demand and live streaming, for new augmented reality and virtual reality experiences. As in video streaming, hologram streaming must support network adaptivity and fast startup, but must also handle large bandwidths, multiple simultaneously streaming objects, and frequent user interaction, which requires a low delay. In this paper, we introduce the first system to our knowledge designed specifically for streaming volumetric media. The system reduces bandwidth by introducing 3D tiles and culling them or reducing their level of detail depending on their relation to the user's view frustum and distance to the user. Our system reduces latency by introducing a window-based buffer, which in contrast to a queue-based buffer allows insertions near the head of the buffer rather than only at the tail of the buffer, to respond quickly to user interaction. To allocate bits between different tiles across multiple objects, we introduce a simple greedy yet provably optimal algorithm for rate-utility optimization. We introduce utility measures based not only on the underlying quality of the representation, but on the level of detail relative to the user's viewpoint and device resolution. The simulation results show that the proposed algorithm provides a superior quality compared with the existing video-streaming approaches adapted to hologram streaming, in terms of utility and user experience over variable, throughput-constrained networks.