The Twist Tensor Nuclear Norm for Video Completion

• Published in: IEEE transaction on neural networks and learning systems Vol. 28; no. 12; pp. 2961 - 2973
• Main Authors: Hu, Wenrui, Tao, Dacheng, Zhang, Wensheng, Xie, Yuan, Yang, Yehui
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.12.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cameras; Computational modeling; Correlation; Data processing; Estimation; Fast Fourier transformations; Fourier transforms; Low-rank tensor estimation (LRTE); Matrix decomposition; Panning; Representations; Tensile stress; tensor multirank; tensor nuclear norm (TNN); twist tensor; video completion; Video sequences
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2016.2611525

In this paper, we propose a new low-rank tensor model based on the circulant algebra, namely, twist tensor nuclear norm (t-TNN). The twist tensor denotes a three-way tensor representation to laterally store 2-D data slices in order. On one hand, t-TNN convexly relaxes the tensor multirank of the twist tensor in the Fourier domain, which allows an efficient computation using fast Fourier transform. On the other, t-TNN is equal to the nuclear norm of block circulant matricization of the twist tensor in the original domain, which extends the traditional matrix nuclear norm in a block circulant way. We test the t-TNN model on a video completion application that aims to fill missing values and the experiment results validate its effectiveness, especially when dealing with video recorded by a nonstationary panning camera. The block circulant matricization of the twist tensor can be transformed into a circulant block representation with nuclear norm invariance. This representation, after transformation, exploits the horizontal translation relationship between the frames in a video, and endows the t-TNN model with a more powerful ability to reconstruct panning videos than the existing state-of-the-art low-rank models.