Empirical study of the modulus as activation function in computer vision applications

In this work we propose a new non-monotonic activation function: the modulus. The majority of the reported research on nonlinearities is focused on monotonic functions. We empirically demonstrate how by using the modulus activation function on computer vision tasks the models generalize better than...

Full description

Saved in:
Bibliographic Details
Published inEngineering applications of artificial intelligence Vol. 120; p. 105863
Main Authors Vallés-Pérez, Iván, Soria-Olivas, Emilio, Martínez-Sober, Marcelino, Serrano-López, Antonio J., Vila-Francés, Joan, Gómez-Sanchís, Juan
Format Journal Article
LanguageEnglish
Published Elsevier Ltd 01.04.2023
Subjects
Activation functions
Deep learning
Optimization
Deep learning
Optimization
Activation functions
Online AccessGet full text

Cover

More Information
Summary:In this work we propose a new non-monotonic activation function: the modulus. The majority of the reported research on nonlinearities is focused on monotonic functions. We empirically demonstrate how by using the modulus activation function on computer vision tasks the models generalize better than with other nonlinearities — up to a 15% accuracy increase in CIFAR100 and 4% in CIFAR10, relative to the best of the benchmark activations tested. With the proposed activation function the vanishing gradient and dying neurons problems disappear, because the derivative of the activation function is always 1 or −1. The simplicity of the proposed function and its derivative make this solution specially suitable for TinyML and hardware applications.
ISSN:0952-1976
1873-6769
DOI:10.1016/j.engappai.2023.105863