Replacing softmax with ReLU in Vision Transformers

• Main Authors: Wortsman, Mitchell, Lee, Jaehoon, Gilmer, Justin, Kornblith, Simon
• Format: Journal Article
• Language: English
• Published: 15.09.2023
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Learning

Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that this degradation is mitigated when dividing by sequence length. Our experiments training small to large vision transform...