A deep learning approach to building an intelligent video surveillance system

• Published in: Multimedia tools and applications Vol. 80; no. 4; pp. 5495 - 5515
• Main Author: Xu, Jie
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.02.2021; Springer Nature B.V
• Subjects: Complex systems; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Deep learning; Face recognition; Facial recognition technology; Multimedia Information Systems; Object recognition; Salience; Special Purpose and Application-Based Systems; Surveillance; Surveillance systems; Telematics; Deep learning; Video surveillance; Face recognition; Machine learning; Object detection
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-020-09964-6

Recent advances in the field of object detection and face recognition have made it possible to develop practical video surveillance systems with embedded object detection and face recognition functionalities that are accurate and fast enough for commercial uses. In this paper, we compare some of the latest approaches to object detection and face recognition and provide reasons why they may or may not be amongst the best to be used in video surveillance applications in terms of both accuracy and speed. It is discovered that Faster R-CNN with Inception ResNet V2 is able to achieve some of the best accuracies while maintaining real-time rates. Single Shot Detector (SSD) with MobileNet, on the other hand, is incredibly fast and still accurate enough for most applications. As for face recognition, FaceNet with Multi-task Cascaded Convolutional Networks (MTCNN) achieves higher accuracy than advances such as DeepFace and DeepID2+ while being faster. An end-to-end video surveillance system is also proposed which could be used as a starting point for more complex systems. Various experiments have also been attempted on trained models with observations explained in detail. We finish by discussing video object detection and video salient object detection approaches which could potentially be used as future improvements to the proposed system.