Diagnostic accuracy of an artificial intelligence online engine in migraine: A multi‐center study

• Published in: Headache Vol. 62; no. 7; pp. 870 - 882
• Main Authors: Cowan, Robert P., Rapoport, Alan M., Blythe, Jim, Rothrock, John, Knievel, Kerry, Peretz, Addie M., Ekpo, Elizabeth, Sanjanwala, Bharati M., Woldeamanuel, Yohannes W.
• Format: Journal Article
• Language: English
• Published: United States Wiley Subscription Services, Inc 01.07.2022
• Subjects: Accuracy; Algorithms; Artificial intelligence; Confidence intervals; Diagnosis; diagnostic accuracy study; Headache; Headaches; Likelihood ratio; Migraine; online engine; Questionnaires; semi‐structured interview; Sensitivity; diagnostic accuracy study; diagnosis; online engine; migraine; semi-structured interview; artificial intelligence
• ISSN: 0017-8748; 1526-4610
• DOI: 10.1111/head.14324

Objective This study assesses the concordance in migraine diagnosis between an online, self‐administered, Computer‐based, Diagnostic Engine (CDE) and semi‐structured interview (SSI) by a headache specialist, both using International Classification of Headache Disorders, 3rd edition (ICHD‐3) criteria. Background Delay in accurate diagnosis is a major barrier to headache care. Accurate computer‐based algorithms may help reduce the need for SSI‐based encounters to arrive at correct ICHD‐3 diagnosis. Methods Between March 2018 and August 2019, adult participants were recruited from three academic headache centers and the community via advertising to our cross‐sectional study. Participants completed two evaluations: phone interview conducted by headache specialists using the SSI and a web‐based expert questionnaire and analytics, CDE. Participants were randomly assigned to either the SSI followed by the web‐based questionnaire or the web‐based questionnaire followed by the SSI. Participants completed protocols a few minutes apart. The concordance in migraine/probable migraine (M/PM) diagnosis between SSI and CDE was measured using Cohen’s kappa statistics. The diagnostic accuracy of CDE was assessed using the SSI as reference standard. Results Of the 276 participants consented, 212 completed both SSI and CDE (study completion rate = 77%; median age = 32 years [interquartile range: 28–40], female:male ratio = 3:1). Concordance in M/PM diagnosis between SSI and CDE was: κ = 0.83 (95% confidence interval [CI]: 0.75–0.91). CDE diagnostic accuracy: sensitivity = 90.1% (118/131), 95% CI: 83.6%–94.6%; specificity = 95.8% (68/71), 95% CI: 88.1%–99.1%. Positive and negative predictive values = 97.0% (95% CI: 91.3%–99.0%) and 86.6% (95% CI: 79.3%–91.5%), respectively, using identified migraine prevalence of 60%. Assuming a general migraine population prevalence of 10%, positive and negative predictive values were 70.3% (95% CI: 43.9%–87.8%) and 98.9% (95% CI: 98.1%–99.3%), respectively. Conclusion The SSI and CDE have excellent concordance in diagnosing M/PM. Positive CDE helps rule in M/PM, through high specificity and positive likelihood ratio. A negative CDE helps rule out M/PM through high sensitivity and low negative likelihood ratio. CDE that mimics SSI logic is a valid tool for migraine diagnosis.