Marina Adriana MERCIONI, Public Dissertation of PhD Thesis

Thesis title: "Improving performance of deep neural networks by developing novel activation functions"

PhD Board:

President: Professor Dr. Eng. Marius MARCU (Politehnica University of Timisoara)

Scientific leader: Professor Dr. Eng. Ștefan HOLBAN (Politehnica  University of Timișoara)

Referees:

Professor Dr. Eng. Rodica POTOLEA (Technical University of Cluj-Napoca)

Professor Dr. Eng. Alexandru CICORTAS (West University of Timișoara)

Professor Dr. Eng. Cătălin-Daniel CĂLEANU (Politehnica  University of Timișoara)

This thesis aims to introduce new activation functions in Deep Learning in order to improve the performance of artificial neural network architectures. Over time, activation functions have developed, but they have inconsistencies, and in many cases lead to large errors on the test set. State-of-the-art comprises a total number of 51 activation functions studied, and the functions proposed in the thesis are both simple functions and composed functions, both with predefined parameters and with learnable parameters. Their efficiency was tested on more than 24 of datasets including data from Computer Vision and Natural Language Processing, but also time series data. More than 10 architectures were used, from simple (LeNet-5) to complex (residual architectures, block architectures, network architecture, so on). I compared my proposals with classic activation functions (sigmoid, hyperbolic tangent), but also with current activation functions such as Rectified Linear Unit, in short RELU, Parametric RELU, Leaky RELU, Softplus, Swish, Mish, Talu, TanhExp, exponential function (ELU), SELU, the proposed functions are leading to performance improvements.