The language of proteins: NLP, machine learning & protein sequences

• Published in: Computational and structural biotechnology journal Vol. 19; pp. 1750 - 1758
• Main Authors: Ofer, Dan, Brandes, Nadav, Linial, Michal
• Format: Journal Article
• Language: English
• Published: Netherlands Elsevier B.V 01.01.2021; Research Network of Computational and Structural Biotechnology; Elsevier
• Subjects: amino acids; Artificial neural networks; automation; Bag of words; BERT; Bioinformatics; biotechnology; computer science; Contextualized embedding; Deep learning; Language models; Natural language processing; Review; Tokenization; Transformer; Word embedding; Word2vec; Deep learning; Contextualized embedding; Word2vec; Bag of words; Transformer; Artificial neural networks; Natural language processing; BERT; Tokenization; Word embedding; Language models; Bioinformatics
• ISSN: 2001-0370; 2001-0370
• DOI: 10.1016/j.csbj.2021.03.022

Natural language processing (NLP) is a field of computer science concerned with automated text and language analysis. In recent years, following a series of breakthroughs in deep and machine learning, NLP methods have shown overwhelming progress. Here, we review the success, promise and pitfalls of applying NLP algorithms to the study of proteins. Proteins, which can be represented as strings of amino-acid letters, are a natural fit to many NLP methods. We explore the conceptual similarities and differences between proteins and language, and review a range of protein-related tasks amenable to machine learning. We present methods for encoding the information of proteins as text and analyzing it with NLP methods, reviewing classic concepts such as bag-of-words, k-mers/n-grams and text search, as well as modern techniques such as word embedding, contextualized embedding, deep learning and neural language models. In particular, we focus on recent innovations such as masked language modeling, self-supervised learning and attention-based models. Finally, we discuss trends and challenges in the intersection of NLP and protein research.