Spatio-temporal adversarial learning for detecting unseen falls

• Published in: Pattern analysis and applications : PAA Vol. 24; no. 1; pp. 381 - 391
• Main Authors: Khan, Shehroz S., Nogas, Jacob, Mihailidis, Alex
• Format: Journal Article
• Language: English
• Published: London Springer London 01.02.2021; Springer Nature B.V
• Subjects: Cameras; Classifiers; Computer Science; Convolution; Fall detection; Frames (data processing); Industrial and Commercial Application; Machine learning; Pattern Recognition; Training; Spatio-temporal; Fall; Depth camera; Adversarial learning; Autoencoder; Thermal camera
• ISSN: 1433-7541; 1433-755X
• DOI: 10.1007/s10044-020-00901-9

Fall detection is an important problem from both the health and machine learning perspective. A fall can lead to severe injuries, long-term impairments or even death in some cases. In terms of machine learning, it presents a severely class imbalance problem with very few or no training data for falls owing to the fact that falls occur rarely. In this paper, we take an alternate philosophy to detect falls in the absence of their training data, by training the classifier on only the normal activities (that are available in abundance) and identifying a fall as an anomaly. To realize such a classifier, we use an adversarial learning framework, which comprises of a spatio-temporal autoencoder for reconstructing input video frames and a spatio-temporal convolution network to discriminate them against original video frames. 3D convolutions are used to learn spatial and temporal features from the input video frames. The adversarial learning of the spatio-temporal autoencoder will enable reconstructing the normal activities of daily living efficiently, thus rendering detecting unseen falls plausible within this framework. We tested the performance of the proposed framework on camera-sensing modalities that may preserve an individual’s privacy (fully or partially), such as thermal and depth camera. Our results on three publicly available datasets show that the proposed spatio-temporal adversarial framework performed better than other baseline frame-based (or spatial) adversarial learning methods.