Skeleton-based human action evaluation using graph convolutional network for monitoring Alzheimer’s progression

• Published in: Pattern recognition Vol. 119; p. 108095
• Main Authors: Yu, Bruce X.B., Liu, Yan, Chan, Keith C.C., Yang, Qintai, Wang, Xiaoying
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.11.2021
• Subjects: Abnormality detection; Alzheimer’s disease; Graph neural network; Human action evaluation; Abnormality detection; Human action evaluation; Alzheimer’s disease; Graph neural network
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2021.108095

•We propose a novel two-task graph convolutional network (2T-GCN) to represent skeleton data for human action evaluation (HAE) tasks of abnormality detection and quality evaluation. To the best of our knowledge, this is the first work that applies GCN to skeleton-based HAE.•We validate the effectiveness of our 2T-GCN on a public dataset, UI-PRMD. Results show that our method outperforms existing methods. Additionally, the Kinect v2 sensor may be more capable of HAE tasks than the Vicon optical tracking system.•We use Kinect v2 to collect an exercise dataset from subjects with Alzheimer’s disease (AD). Based on experiments, findings show that our method can be effective for HAE tasks. We also observe that the evaluation scores for some exercises coincide with clinical evaluations of AD. Human action evaluation (HAE) involves judgments about the abnormality and quality of human actions. If performed effectively, HAE based on skeleton data can be used to monitor the outcomes of behavioral therapies for Alzheimer’s disease (AD). In this paper, we propose a two-task graph convolutional network (2T-GCN) to represent skeleton data for HAE tasks involving abnormality detection and quality evaluation. The network is first evaluated using the UI-PRMD dataset and demonstrates accurate abnormality detection. Regarding quality evaluation, in addition to laboratory-collected UI-PRMD data, we test the network on a set of real exercise data collected from patients with AD. A numerical score indicating the degree to which actions deviate from normal is taken to reflect the severity of AD; thus, we apply 2T-GCN to determine such scores. Experimental results show that numerical scores for certain exercises performed by patients with AD are consistent with their AD severity level as identified by clinical staff. This corroboration highlights the potential of our approach for monitoring AD and other neurodegenerative diseases.