TinyViT: Fast Pretraining Distillation for Small Vision Transformers

• Published in: arXiv.org
• Main Authors: Wu, Kan, Zhang, Jinnian, Houwen Peng, Liu, Mengchen, Xiao, Bin, Fu, Jianlong, Lu, Yuan
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 21.07.2022
• Subjects: Accuracy; Computation; Computer vision; Constraint modelling; Distillation; Knowledge management; Parameters

Vision transformer (ViT) recently has drawn great attention in computer vision due to its remarkable model capability. However, most prevailing ViT models suffer from huge number of parameters, restricting their applicability on devices with limited resources. To alleviate this issue, we propose Tin...