Eye Fixation Assisted Video Saliency Detection via Total Variation-Based Pairwise Interaction

• Published in: IEEE transactions on image processing Vol. 27; no. 10; pp. 4724 - 4739
• Main Authors: Qiu, Wenliang, Gao, Xinbo, Han, Bing
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2018
• Subjects: conditional random fields; Data models; eye fixation; Gaze tracking; geodesic distance; Image color analysis; Optical imaging; pairwise interaction; Saliency detection; total variation; Tracking; Video saliency detection; Visualization; weakly supervised learning
• ISSN: 1057-7149; 1941-0042; 1941-0042
• DOI: 10.1109/TIP.2018.2843680

As human visual attention is naturally biased toward foreground objects in a scene, it can be used to extract salient objects in video clips. In this paper, we proposed a weakly supervised learning-based video saliency detection algorithm utilizing eye fixations information from multiple subjects. Our main idea is to extend eye fixations to saliency regions step by step. First, visual seeds are collected using multiple color space geodesic distance-based seed region mapping with filtered and extended eye fixations. This operation helps raw fixation points spread to the most likely salient regions, namely, visual seed regions. Second, in order to seize the essential scene structure from video sequences, we introduce the total variance-based pairwise interaction model to learn the potential pairwise relationship between foreground and background within a frame or across video frames. In this vein, visual seed regions eventually grow into salient regions. Compared with previous approaches the generated saliency maps have two most outstanding properties: integrity and purity, which are conductive to segment the foreground and significant to the follow-up tasks. Extensive quantitative and qualitative experiments on various video sequences demonstrate that the proposed method outperforms the state-of-the-art image and video saliency detection algorithms.