Masked inverse folding with sequence transfer for protein representation learning

• Published in: Protein engineering, design and selection Vol. 36
• Main Authors: Yang, Kevin K, Zanichelli, Niccolò, Yeh, Hugh
• Format: Journal Article
• Language: English
• Published: Oxford University Press 21.01.2023
• Subjects: machine learning; pretraining
• ISSN: 1741-0126; 1741-0134; 1741-0134
• DOI: 10.1093/protein/gzad015

Abstract Self-supervised pretraining on protein sequences has led to state-of-the art performance on protein function and fitness prediction. However, sequence-only methods ignore the rich information contained in experimental and predicted protein structures. Meanwhile, inverse folding methods reconstruct a protein’s amino-acid sequence given its structure, but do not take advantage of sequences that do not have known structures. In this study, we train a masked inverse folding protein masked language model parameterized as a structured graph neural network. During pretraining, this model learns to reconstruct corrupted sequences conditioned on the backbone structure. We then show that using the outputs from a pretrained sequence-only protein masked language model as input to the inverse folding model further improves pretraining perplexity. We evaluate both of these models on downstream protein engineering tasks and analyze the effect of using information from experimental or predicted structures on performance.