Compact Temporal Trajectory Representation for Talking Face Video Compression

• Published in: IEEE transactions on circuits and systems for video technology Vol. 33; no. 11; p. 1
• Main Authors: Chen, Bolin, Wang, Zhao, Li, Binzhe, Wang, Shiqi, Ye, Yan
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.11.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Coding; Coding standards; compact feature representation; Encoding; Faces; Frames (data processing); Image coding; Nonlinear dynamics; Object oriented modeling; Representations; Talking; Talking face; Trajectory; Video coding; Video communication; Video compression; visual quality assessment
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2023.3271130

In this paper, we propose to compactly represent the nonlinear dynamics along the temporal trajectories for talking face video compression. By projecting the frames into a high dimensional space, the temporal trajectories of talking face frames, which are complex, non-linear and difficult to extrapolate, are implicitly modelled in an end-to-end inference framework based upon very compact feature representation. As such, the proposed framework is suitable for ultra-low bandwidth video communication and can guarantee the quality of the reconstructed video in such applications. The proposed compression scheme is also robust against large head-pose motions, due to the delicately designed dynamic reference refresh and temporal stabilization mechanisms. Experimental results demonstrate that compared to the state-of-the-art video coding standard Versatile Video Coding (VVC) as well as the latest generative compression schemes, our proposed scheme is superior in terms of both objective and subjective quality at the same bitrate. The project page can be found at https://github.com/Berlin0610/CTTR.