Large-scale foundation model on single-cell transcriptomics

• Published in: Nature methods Vol. 21; no. 8; pp. 1481 - 1491
• Main Authors: Hao, Minsheng, Gong, Jing, Zeng, Xin, Liu, Chiming, Guo, Yucheng, Cheng, Xingyi, Wang, Taifeng, Ma, Jianzhu, Zhang, Xuegong, Song, Le
• Format: Journal Article
• Language: English
• Published: New York Nature Publishing Group 01.08.2024
• Subjects: Annotations; Cell culture; Cell size; Design parameters; Gene expression; Genes; Medical research; Natural language processing; Scale models; Task complexity; Transcriptomics
• ISSN: 1548-7091; 1548-7105; 1548-7105
• DOI: 10.1038/s41592-024-02305-7

Large pretrained models have become foundation models leading to breakthroughs in natural language processing and related fields. Developing foundation models for deciphering the ‘languages’ of cells and facilitating biomedical research is promising yet challenging. Here we developed a large pretrained model scFoundation, also named ‘xTrimoscFoundationα’, with 100 million parameters covering about 20,000 genes, pretrained on over 50 million human single-cell transcriptomic profiles. scFoundation is a large-scale model in terms of the size of trainable parameters, dimensionality of genes and volume of training data. Its asymmetric transformer-like architecture and pretraining task design empower effectively capturing complex context relations among genes in a variety of cell types and states. Experiments showed its merit as a foundation model that achieved state-of-the-art performances in a diverse array of single-cell analysis tasks such as gene expression enhancement, tissue drug response prediction, single-cell drug response classification, single-cell perturbation prediction, cell type annotation and gene module inference.scFoundation, with 100 million parameters covering about 20,000 genes, pretrained on over 50 million single-cell transcriptomics profiles, is a foundation model for diverse tasks of single-cell analysis.