A Novel Key-Frames Selection Framework for Comprehensive Video Summarization

• Published in: IEEE transactions on circuits and systems for video technology Vol. 30; no. 2; pp. 577 - 589
• Main Authors: Huang, Cheng, Wang, Hongmei
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.02.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Capsules Net; Deep learning; Feature extraction; Frames (data processing); Image segmentation; Key-frames selection; Motion segmentation; Optical flow; Optical flow (image analysis); self-attention; Spatiotemporal phenomena; Streaming media; transition effects detection; Video data; video summarization; Visualization
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2019.2890899

Video summarization (VSUMM) has become a popular method in processing massive video data. The key point of VSUMM is to select the key frames to represent the effective contents of a video sequence. The existing methods can only extract the static images of videos as the content summarization, but they ignore the representation of motion information. To cope with these issues, a novel framework for an efficient video content summarization as well as video motion summarization is proposed. Initially, Capsules Net is trained as a spatiotemporal information extractor, and an inter-frames motion curve is generated based on those spatiotemporal features. Subsequently, a transition effects detection method is proposed to automatically segment the video streams into shots. Finally, a self-attention model is introduced to select key-frames sequences inside the shots; thus, key static images are selected as video content summarization, and optical flows can be calculated as video motion summarization. The ultimate experimental results demonstrate that our method is competitive on VSUMM, TvSum, SumMe, and RAI datasets about shot segmentation and video content summarization, and can also represent a good motion summarization result.