Evaluation of the accuracy of an artificial intelligence in identifying contraindications to exercise therapy - Comparison with and interrater reliability of physical therapists judgments

• Published in: Health and technology Vol. 14; no. 3; pp. 513 - 522
• Main Authors: Annika, Griefahn, Christoff, Zalpour, Kerstin, Luedtke
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.05.2024; Springer Nature B.V
• Subjects: Accuracy; Agreements; Artificial intelligence; Back pain; Biological and Medical Physics; Biomedical Engineering and Bioengineering; Biomedicine; Biophysics; Computational Biology/Bioinformatics; Contraindications; Datasets; Engineering; Exercise; Expert systems; Medicine/Public Health; Musculoskeletal diseases; Original Paper; Patient satisfaction; Performance evaluation; Physical fitness; Physical therapists; Physical therapy; R & D/Technology Policy; Sensitivity analysis; Statistical analysis; Therapy; Validation; Accuracy; Contraindication; Artificial intelligence; Exercise therapy
• ISSN: 2190-7188; 2190-7196
• DOI: 10.1007/s12553-024-00827-w

Purpose The study validates a rule-based system for identifying contraindications to exercise therapy in a medical context. It evaluates accuracy and performance by comparing it with physical therapists’ assessments and patients' characteristics. Method The dataset included 80 patient cases with clinical characteristics assessed by 20 physical therapists for contraindications to exercise therapy. Fleiss kappa and pooled kappa values measured agreement between physical therapists and AI. AI performance was assessed by sensitivity, specificity, accuracy and F1 score. Clinical characteristics were compared between therapists' votes using ANOVA and Bonferroni post-hoc test. Results The physical therapists had a mean age of 40.85 (8.23) years and a mean experience of 14.53 (8.20) years. Out of 64 patient cases, there was consensus on 35 cases with no contraindication and 29 cases with a consensus on “contraindication exists” for exercise therapy. In 16 cases there was no consensus between therapists. Overall, therapists had 87.5% agreement with Fleiss Kappa κ π  = .43. The pooled kappa value between therapists and AI was κ pooled  = .63. AI achieved perfect values (1) for sensitivity, specificity, accuracy and F1 score. Statistically, consensus-based comparisons by therapists revealed significant differences in pain intensity, duration, timing, and quality. Conclusion The study shows significant agreement between physical therapists and the AI, consistent with similar musculoskeletal studies. Various clinical characteristics highlight the importance of clinical reasoning and contraindication detection. In conclusion, advanced technologies such as decision support and expert systems could have a profound impact on clinical practice, improving accuracy, personalized exercises and telemedicine referrals for efficient care and improved patient decisions. Trial registration 30.12.2021 via OSF Registries, https://doi.org/10.17605/OSF.IO/YCNJQ .