ErfReLU: adaptive activation function for deep neural network

• Published in: Pattern analysis and applications : PAA Vol. 27; no. 2
• Main Authors: Rajanand, Ashish, Singh, Pradeep
• Format: Journal Article
• Language: English
• Published: London Springer London 01.06.2024; Springer Nature B.V
• Subjects: Artificial neural networks; Computer Science; Deep learning; Error functions; Machine learning; Pattern Recognition; Performance evaluation; Theoretical Advances; Deep learning; Activation function; Image classification
• ISSN: 1433-7541; 1433-755X
• DOI: 10.1007/s10044-024-01277-w

Recent research has found that the activation function (AF) plays a significant role in introducing non-linearity to enhance the performance of deep learning networks. Researchers recently started developing activation functions that can be trained throughout the learning process, known as trainable, or adaptive activation functions (AAF). Research on AAF that enhances the outcomes is still in its early stages. In this paper, a novel activation function ‘ErfReLU’ has been developed based on the erf function and ReLU. This function leverages the advantages of both the Rectified Linear Unit (ReLU) and the error function (erf). A comprehensive overview of activation functions like Sigmoid, ReLU, Tanh, and their properties have been briefly explained. Adaptive activation functions like Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf is also presented. Lastly, comparative performance analysis of 9 trainable activation functions namely Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf with the proposed one has been performed. These activation functions are used in MobileNet, VGG16, and ResNet models and their performance is evaluated on benchmark datasets such as CIFAR-10, MNIST, and FMNIST.