Web Video Event Recognition by Semantic Analysis From Ubiquitous Documents

• Published in: IEEE transactions on image processing Vol. 25; no. 12; pp. 5689 - 5701
• Main Authors: Yu, Litao, Yang, Yang, Huang, Zi, Wang, Peng, Song, Jingkuan, Shen, Heng Tao
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.12.2016
• Subjects: Adaptation models; event knowledge base; Image recognition; Knowledge based systems; Multimedia communication; Semantics; Streaming media; two-view adaptive regression; Video event recognition; Visualization
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2016.2614136

In recent years, the task of event recognition from videos has attracted increasing interest in multimedia area. While most of the existing research was mainly focused on exploring visual cues to handle relatively small-granular events, it is difficult to directly analyze video content without any prior knowledge. Therefore, synthesizing both the visual and semantic analysis is a natural way for video event understanding. In this paper, we study the problem of Web video event recognition, where Web videos often describe large-granular events and carry limited textual information. Key challenges include how to accurately represent event semantics from incomplete textual information and how to effectively explore the correlation between visual and textual cues for video event understanding. We propose a novel framework to perform complex event recognition from Web videos. In order to compensate the insufficient expressive power of visual cues, we construct an event knowledge base by deeply mining semantic information from ubiquitous Web documents. This event knowledge base is capable of describing each event with comprehensive semantics. By utilizing this base, the textual cues for a video can be significantly enriched. Furthermore, we introduce a two-view adaptive regression model, which explores the intrinsic correlation between the visual and textual cues of the videos to learn reliable classifiers. Extensive experiments on two real-world video data sets show the effectiveness of our proposed framework and prove that the event knowledge base indeed helps improve the performance of Web video event recognition.