Expert video-surveillance system for real-time detection of suspicious behaviors in shopping malls

• Published in: Expert systems with applications Vol. 42; no. 21; pp. 7991 - 8005
• Main Authors: Arroyo, Roberto, Yebes, J. Javier, Bergasa, Luis M., Daza, Iván G., Almazán, Javier
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 30.11.2015
• Subjects: Appearance features; Background subtraction; Behavioral analysis; Human behavior; Human tracking; Occlusion; Occlusion management; Shopping malls; State of the art; Surface layer; Texture; Tracking; Trajectories; Video-surveillance in shopping malls; Appearance features; Occlusion management; Background subtraction; Behavioral analysis; Video-surveillance in shopping malls; Human tracking
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2015.06.016

•Tracking-by-detection based on segmentation, Kalman predictions and LSAP association.•Occlusion management: SVM kernel metric for GCH+LBP+HOG image features.•Overall performance near to 85% while tracking under occlusions in CAVIAR dataset.•Human behavior analysis (exits, loitering, etc.) in naturalistic scenes in shops.•Real-time multi-camera performance with a processing capacity near to 50fps/camera. Expert video-surveillance systems are a powerful tool applied in varied scenarios with the aim of automatizing the detection of different risk situations and helping human security officers to take appropriate decisions in order to enhance the protection of assets. In this paper, we propose a complete expert system focused on the real-time detection of potentially suspicious behaviors in shopping malls. Our video-surveillance methodology contributes several innovative proposals that compose a robust application which is able to efficiently track the trajectories of people and to discover questionable actions in a shop context. As a first step, our system applies an image segmentation to locate the foreground objects in scene. In this case, the most effective background subtraction algorithms of the state of the art are compared to find the most suitable for our expert video-surveillance application. After the segmentation stage, the detected blobs may represent full or partial people bodies, thus, we have implemented a novel blob fusion technique to group the partial blobs into the final human targets. Then, we contribute an innovative tracking algorithm which is not only based on people trajectories as the most part of state-of-the-art methods, but also on people appearance in occlusion situations. This tracking is carried out employing a new two-step method: (1) the detections-to-tracks association is solved by using Kalman filtering combined with an own-designed cost optimization for the Linear Sum Assignment Problem (LSAP); and (2) the occlusion management is based on SVM kernels to compute distances between appearance features such as GCH, LBP and HOG. The application of these three features for recognizing human appearance provides a great performance compared to other description techniques, because color, texture and gradient information are effectively combined to obtain a robust visual description of people. Finally, the resultant trajectories of people obtained in the tracking stage are processed by our expert video-surveillance system for analyzing human behaviors and identifying potential shopping mall alarm situations, as are shop entry or exit of people, suspicious behaviors such as loitering and unattended cash desk situations. With the aim of evaluating the performance of some of the main contributions of our proposal, we use the publicly available CAVIAR dataset for testing the proposed tracking method with a success near to 85% in occlusion situations. According to this performance, we corroborate in the presented results that the precision and efficiency of our tracking method is comparable and slightly superior to the most recent state-of-the-art works. Furthermore, the alarms given off by our application are evaluated on a naturalistic private dataset, where it is evidenced that our expert video-surveillance system can effectively detect suspicious behaviors with a low computational cost in a shopping mall context.