Community challenges in biomedical text mining over 10 years: success, failure and the future

• Published in: Briefings in bioinformatics Vol. 17; no. 1; pp. 132 - 144
• Main Authors: Huang, Chung-Chi, Lu, Zhiyong
• Format: Journal Article
• Language: English
• Published: England Oxford Publishing Limited (England) 01.01.2016; Oxford University Press
• Subjects: Algorithms; Bioinformatics; Biomedical research; Computational Biology - methods; Computational Biology - trends; Data mining; Data Mining - methods; Data Mining - trends; Databases, Genetic - statistics & numerical data; Humans; medicine; Natural Language Processing; prediction; protein structure; Proteins; stakeholders; critical assessment; BioNLP shared tasks; text mining; BioNLP challenges; biomedical natural language processing (BioNLP)
• ISSN: 1467-5463; 1477-4054; 1477-4054
• DOI: 10.1093/bib/bbv024

One effective way to improve the state of the art is through competitions. Following the success of the Critical Assessment of protein Structure Prediction (CASP) in bioinformatics research, a number of challenge evaluations have been organized by the text-mining research community to assess and advance natural language processing (NLP) research for biomedicine. In this article, we review the different community challenge evaluations held from 2002 to 2014 and their respective tasks. Furthermore, we examine these challenge tasks through their targeted problems in NLP research and biomedical applications, respectively. Next, we describe the general workflow of organizing a Biomedical NLP (BioNLP) challenge and involved stakeholders (task organizers, task data producers, task participants and end users). Finally, we summarize the impact and contributions by taking into account different BioNLP challenges as a whole, followed by a discussion of their limitations and difficulties. We conclude with future trends in BioNLP challenge evaluations.