MG-BERT: leveraging unsupervised atomic representation learning for molecular property prediction

• Published in: Briefings in bioinformatics Vol. 22; no. 6
• Main Authors: Zhang, Xiao-Chen, Wu, Cheng-Kun, Yang, Zhi-Jiang, Wu, Zhen-Xing, Yi, Jia-Cai, Hsieh, Chang-Yu, Hou, Ting-Jun, Cao, Dong-Sheng
• Format: Journal Article
• Language: English
• Published: England 05.11.2021
• Subjects: Models, Theoretical; Neural Networks, Computer; atomic representation; deep learning; self-supervised learning; molecular graph BERT; molecular property prediction
• ISSN: 1467-5463; 1477-4054; 1477-4054
• DOI: 10.1093/bib/bbab152

Motivation: Accurate and efficient prediction of molecular properties is one of the fundamental issues in drug design and discovery pipelines. Traditional feature engineering-based approaches require extensive expertise in the feature design and selection process. With the development of artificial intelligence (AI) technologies, data-driven methods exhibit unparalleled advantages over the feature engineering-based methods in various domains. Nevertheless, when applied to molecular property prediction, AI models usually suffer from the scarcity of labeled data and show poor generalization ability. Results: In this study, we proposed molecular graph BERT (MG-BERT), which integrates the local message passing mechanism of graph neural networks (GNNs) into the powerful BERT model to facilitate learning from molecular graphs. Furthermore, an effective self-supervised learning strategy named masked atoms prediction was proposed to pretrain the MG-BERT model on a large amount of unlabeled data to mine context information in molecules. We found the MG-BERT model can generate context-sensitive atomic representations after pretraining and transfer the learned knowledge to the prediction of a variety of molecular properties. The experimental results show that the pretrained MG-BERT model with a little extra fine-tuning can consistently outperform the state-of-the-art methods on all 11 ADMET datasets. Moreover, the MG-BERT model leverages attention mechanisms to focus on atomic features essential to the target property, providing excellent interpretability for the trained model. The MG-BERT model does not require any hand-crafted feature as input and is more reliable due to its excellent interpretability, providing a novel framework to develop state-of-the-art models for a wide range of drug discovery tasks.