Neural Network with Smooth Activation Functions and without Bottlenecks Is Almost Surely a Morse Function

Modern approaches in deep neural networks have a number of issues related to the learning process and computational costs. This article considers the architecture grounded on an alternative approach to the basic unit of the neural network. This approach achieves optimization in the calculations and gives rise to an alternative way to solve the problems of the vanishing and exploding gradient. The main issue of the article is the usage of the deep stacked neo-fuzzy system, which uses a generalized neo-fuzzy neuron to optimize the learning process. This approach is non-standard from a theoretical point of view, so the paper presents the necessary mathematical calculations and describes all the intricacies of using this architecture from a practical point of view. From a theoretical point, the network learning process is fully disclosed. Derived all necessary calculations for the use of the backpropagation algorithm for network training. A feature of the network is the rapid calculation of the derivative for the activation functions of neurons. This is achieved through the use of fuzzy membership functions. The paper shows that the derivative of such function is a constant, and this is a reason for the statement of increasing in the optimization rate in comparison with neural networks which use neurons with more common activation functions (ReLU, sigmoid). The paper highlights the main points that can be improved in further theoretical developments on this topic. In general, these issues are related to the calculation of the activation function. The proposed methods cope with these points and allow approximation using the network, but the authors already have theoretical justifications for improving the speed and approximation properties of the network. The results of the comparison of the proposed network with standard neural network architectures are shown

Download Full-text

Hybrid Additive-Multiplicative Evolutionary Neural Network Architectures Employing Heterogeneous Activation Functions

Journal of Japan Society for Fuzzy Theory and Intelligent Informatics ◽

10.3156/jsoft.16.102_2 ◽

2004 ◽

Vol 16 (1) ◽

pp. 102

Author(s):

Eduardo Masato Iyoda

Keyword(s):

Neural Network ◽

Network Architectures ◽

Activation Functions ◽

Evolutionary Neural Network ◽

Neural Network Architectures

Download Full-text

Probabilistic Neural Network With Complex Exponential Activation Functions in Image Recognition

IEEE Transactions on Neural Networks and Learning Systems ◽

10.1109/tnnls.2019.2908973 ◽

2020 ◽

Vol 31 (2) ◽

pp. 651-660 ◽

Cited By ~ 5

Author(s):

Andrey Savchenko

Keyword(s):

Neural Network ◽

Image Recognition ◽

Probabilistic Neural Network ◽

Activation Functions ◽

Complex Exponential

Download Full-text

An Analysis of State-of-the-art Activation Functions For Supervised Deep Neural Network

10.31219/osf.io/2zk6a ◽

2021 ◽

Author(s):

Anh Nguyen ◽

Khoa Pham ◽

Dat Ngo ◽

Thanh Ngo ◽

Lam Pham

Keyword(s):

Neural Network ◽

Supervised Classification ◽

Deep Neural Network ◽

State Of The Art ◽

Network Architectures ◽

Activation Functions ◽

Scene Classification ◽

Learning Network ◽

Deep Learning Network

This paper provides an analysis of state-of-the-art activation functions with respect to supervised classification of deep neural network. These activation functions comprise of Rectified Linear Units (ReLU), Exponential Linear Unit (ELU), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU), and the Inverse Square Root Linear Unit (ISRLU). To evaluate, experiments over two deep learning network architectures integrating these activation functions are conducted. The first model, basing on Multilayer Perceptron (MLP), is evaluated with MNIST dataset to perform these activation functions.Meanwhile, the second model, likely VGGish-based architecture, is applied for Acoustic Scene Classification (ASC) Task 1A in DCASE 2018 challenge, thus evaluate whether these activation functions work well in different datasets as well as different network architectures.

Download Full-text

Binary and Multiclass Text Classification by Means of Separable Convolutional Neural Network

Inventions ◽

10.3390/inventions6040070 ◽

2021 ◽

Vol 6 (4) ◽

pp. 70

Author(s):

Elena Solovyeva ◽

Ali Abdullah

Keyword(s):

Neural Network ◽

Neural Networks ◽

Convolutional Neural Network ◽

Recurrent Neural Networks ◽

Low Cost ◽

Computational Cost ◽

High Accuracy ◽

Activation Functions ◽

Fully Connected ◽

Fully Connected Networks

In this paper, the structure of a separable convolutional neural network that consists of an embedding layer, separable convolutional layers, convolutional layer and global average pooling is represented for binary and multiclass text classifications. The advantage of the proposed structure is the absence of multiple fully connected layers, which is used to increase the classification accuracy but raises the computational cost. The combination of low-cost separable convolutional layers and a convolutional layer is proposed to gain high accuracy and, simultaneously, to reduce the complexity of neural classifiers. Advantages are demonstrated at binary and multiclass classifications of written texts by means of the proposed networks under the sigmoid and Softmax activation functions in convolutional layer. At binary and multiclass classifications, the accuracy obtained by separable convolutional neural networks is higher in comparison with some investigated types of recurrent neural networks and fully connected networks.

Download Full-text