Hybrid multi-modality multi-task learning for forecasting progression trajectories in subjective cognitive decline

• Published in: Neural networks Vol. 186; p. 107263
• Main Authors: Yu, Minhui, Fang, Yuqi, Liu, Yunbi, Bozoki, Andrea C., Xiao, Shifu, Yue, Ling, Liu, Mingxia
• Format: Journal Article
• Language: English
• Published: United States Elsevier Ltd 01.06.2025
• Subjects: Aged; Cognitive Dysfunction - diagnostic imaging; Cognitive Dysfunction - psychology; Disease Progression; Female; Forecasting - methods; Humans; Machine Learning; Magnetic Resonance Imaging - methods; Male; MRI; Multi-modality fusion; PET; Positron-Emission Tomography - methods; Subjective cognitive decline; Subjective cognitive decline; MRI; PET; Multi-modality fusion
• ISSN: 0893-6080; 1879-2782; 1879-2782
• DOI: 10.1016/j.neunet.2025.107263

While numerous studies strive to exploit the complementary potential of MRI and PET using learning-based methods, the effective fusion of the two modalities remains a tricky problem due to their inherently distinctive properties. In addition, current studies often face the problem of small sample sizes and missing PET data due to factors such as patient withdrawal or low image quality. To this end, we propose a hybrid multi-modality multi-task learning (HM2L) framework with cross-domain knowledge transfer for forecasting trajectories of SCD progression. Our HM2L comprises (1) missing PET imputation, (2) multi-modality feature extraction for MRI and PET feature learning with a novel softmax-triplet constraint, (3) attention-based multi-modality fusion of MRI and PET features, and (4) multi-task prediction of category labels and clinical scores such as Mini-Mental State Examination (MMSE) and Geriatric Depression Scale (GDS). To handle problems with small sample sizes, a transfer learning strategy is developed to enable knowledge transfer from a relatively large scale dataset with MRI and PET from 795 subjects to two small-scale SCD cohorts with a total of 136 subjects. Experimental results indicate HM2L surpasses several state-of-the-art methods in jointly predicting category labels and clinical scores of subjective cognitive decline. Results show that the MMSE scores of SCD subjects who develop mild cognitive impairment during the 2-year/7-year follow-up are significantly lower than those of subjects who remain stable, while there exists a complex relationship between SCD progression with GDS. •Creating a hybrid multi-task learning method for longitudinal disease analysis.•Designing a softmax-triplet constraint aid in feature extraction in latent space.•Developing a transfer learning method to alleviate the small data issue.•Conducting extensive experiments on two cohorts in two prediction tasks.