Time-resolved evaluation of compound repositioning predictions on a text-mined knowledge network

• Published in: BMC bioinformatics Vol. 20; no. 1; pp. 653 - 12
• Main Authors: Mayers, Michael, Li, Tong Shu, Queralt-Rosinach, Núria, Su, Andrew I.
• Format: Journal Article
• Language: English
• Published: England BioMed Central Ltd 11.12.2019; BioMed Central; BMC
• Subjects: Algorithms; Analysis; Artificial intelligence; Compound repositioning; Computational Biology - methods; Computer applications; Computer networks; Data Mining; Disease; Drug central; Drug Repositioning; Drugs; Gene expression; Heterogeneous network; Humans; Knowledge; Knowledge Bases; Machine Learning; Medical Subject Headings-MeSH; Natural language processing; Ontology; Performance measurement; Predictions; Reproducibility of Results; Semantic Medline database; Semantic network; Semantics; Time Factors; Unified medical language system; Compound repositioning; Heterogeneous network; Semantic Medline database; Machine learning; Semantic network; Drug central; Unified medical language system
• ISSN: 1471-2105; 1471-2105
• DOI: 10.1186/s12859-019-3297-0

Computational compound repositioning has the potential for identifying new uses for existing drugs, and new algorithms and data source aggregation strategies provide ever-improving results via in silico metrics. However, even with these advances, the number of compounds successfully repositioned via computational screening remains low. New strategies for algorithm evaluation that more accurately reflect the repositioning potential of a compound could provide a better target for future optimizations. Using a text-mined database, we applied a previously described network-based computational repositioning algorithm, yielding strong results via cross-validation, averaging 0.95 AUROC on test-set indications. However, to better approximate a real-world scenario, we built a time-resolved evaluation framework. At various time points, we built networks corresponding to prior knowledge for use as a training set, and then predicted on a test set comprised of indications that were subsequently described. This framework showed a marked reduction in performance, peaking in performance metrics with the 1985 network at an AUROC of .797. Examining performance reductions due to removal of specific types of relationships highlighted the importance of drug-drug and disease-disease similarity metrics. Using data from future timepoints, we demonstrate that further acquisition of these kinds of data may help improve computational results. Evaluating a repositioning algorithm using indications unknown to input network better tunes its ability to find emerging drug indications, rather than finding those which have been randomly withheld. Focusing efforts on improving algorithmic performance in a time-resolved paradigm may further improve computational repositioning predictions.