Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders
Vision Transformers (ViTs) have become ubiquitous in computer vision. Despite their success, ViTs lack inductive biases, which can make it difficult to train them with limited data. To address this challenge, prior studies suggest training ViTs with self-supervised learning (SSL) and fine-tuning seq...
Saved in:
Main Authors | , , , , , , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
31.10.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Vision Transformers (ViTs) have become ubiquitous in computer vision. Despite
their success, ViTs lack inductive biases, which can make it difficult to train
them with limited data. To address this challenge, prior studies suggest
training ViTs with self-supervised learning (SSL) and fine-tuning sequentially.
However, we observe that jointly optimizing ViTs for the primary task and a
Self-Supervised Auxiliary Task (SSAT) is surprisingly beneficial when the
amount of training data is limited. We explore the appropriate SSL tasks that
can be optimized alongside the primary task, the training schemes for these
tasks, and the data scale at which they can be most effective. Our findings
reveal that SSAT is a powerful technique that enables ViTs to leverage the
unique characteristics of both the self-supervised and primary tasks, achieving
better performance than typical ViTs pre-training with SSL and fine-tuning
sequentially. Our experiments, conducted on 10 datasets, demonstrate that SSAT
significantly improves ViT performance while reducing carbon footprint. We also
confirm the effectiveness of SSAT in the video domain for deepfake detection,
showcasing its generalizability. Our code is available at
https://github.com/dominickrei/Limited-data-vits. |
---|---|
DOI: | 10.48550/arxiv.2310.20704 |