A transformer architecture based on BERT and 2D convolutional neural network to identify DNA enhancers from sequence information

• Published in: Briefings in bioinformatics Vol. 22; no. 5
• Main Authors: Le, Nguyen Quoc Khanh, Ho, Quang-Thai, Nguyen, Trinh-Trung-Duong, Ou, Yu-Yen
• Format: Journal Article
• Language: English
• Published: England 02.09.2021
• Subjects: NLP transformer; biological sequence; BERT; contextualized word embedding; DNA enhancer; convolutional neural network
• ISSN: 1467-5463; 1477-4054; 1477-4054
• DOI: 10.1093/bib/bbab005

Recently, language representation models have drawn a lot of attention in the natural language processing field due to their remarkable results. Among them, bidirectional encoder representations from transformers (BERT) has proven to be a simple, yet powerful language model that achieved novel state-of-the-art performance. BERT adopted the concept of contextualized word embedding to capture the semantics and context of the words in which they appeared. In this study, we present a novel technique by incorporating BERT-based multilingual model in bioinformatics to represent the information of DNA sequences. We treated DNA sequences as natural sentences and then used BERT models to transform them into fixed-length numerical matrices. As a case study, we applied our method to DNA enhancer prediction, which is a well-known and challenging problem in this field. We then observed that our BERT-based features improved more than 5–10% in terms of sensitivity, specificity, accuracy and Matthews correlation coefficient compared to the current state-of-the-art features in bioinformatics. Moreover, advanced experiments show that deep learning (as represented by 2D convolutional neural networks; CNN) holds potential in learning BERT features better than other traditional machine learning techniques. In conclusion, we suggest that BERT and 2D CNNs could open a new avenue in biological modeling using sequence information.