Replacing softmax with ReLU in Vision Transformers

Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that this degradation is mitigated when dividing by sequence length. Our experiments training small to large vision transform...

Full description

Saved in:
Bibliographic Details
Main Authors Wortsman, Mitchell, Lee, Jaehoon, Gilmer, Justin, Kornblith, Simon
Format Journal Article
LanguageEnglish
Published 15.09.2023
Subjects
Computer Science - Computer Vision and Pattern Recognition
Computer Science - Learning
Online AccessGet full text

Cover

More Information
Summary:Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that this degradation is mitigated when dividing by sequence length. Our experiments training small to large vision transformers on ImageNet-21k indicate that ReLU-attention can approach or match the performance of softmax-attention in terms of scaling behavior as a function of compute.
DOI:10.48550/arxiv.2309.08586