A Hierarchical Visual Model for Video Object Summarization

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 32; no. 12; pp. 2178 - 2190
• Main Authors: Liu, D, Gang Hua, Tsuhan Chen
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01.12.2010; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Applied sciences; Artificial intelligence; Clutter; Computer science; control theory; systems; Continuity; Exact sciences and technology; Frames; Graphical models; Internet; Labeling; Layout; Learning; Mathematical models; Multiple Instance Learning; Object detection; Pattern analysis; Pattern recognition. Digital image processing. Computational geometry; probabilistic graphical model; semi-supervised learning; Semisupervised learning; Speech and sound recognition and synthesis. Linguistics; Streaming media; Studies; Surveillance; Temporal logic; Topic model; video object summarization; Visual; video object summarization; Abstract; Video signal; Object recognition; Clutter; Modeling; Supervised learning; Object detection; Linguistics; Bayes network; Hierarchical system; Multiple Instance Learning; semi-supervised learning; Localization; Pattern analysis; Artificial intelligence; probabilistic graphical model; Document analysis; Topic model
• ISSN: 0162-8828; 1939-3539
• DOI: 10.1109/TPAMI.2010.31

We propose a novel method for removing irrelevant frames from a video given user-provided frame-level labeling for a very small number of frames. We first hypothesize a number of windows which possibly contain the object of interest, and then determine which window(s) truly contain the object of interest. Our method enjoys several favorable properties. First, compared to approaches where a single descriptor is used to describe a whole frame, each window's feature descriptor has the chance of genuinely describing the object of interest; hence it is less affected by background clutter. Second, by considering the temporal continuity of a video instead of treating frames as independent, we can hypothesize the location of the windows more accurately. Third, by infusing prior knowledge into the patch-level model, we can precisely follow the trajectory of the object of interest. This allows us to largely reduce the number of windows and hence reduce the chance of overfitting the data during learning. We demonstrate the effectiveness of the method by comparing it to several other semi-supervised learning approaches on challenging video clips.