Signal processing algorithm for neural networks with integrodifferential splines as an activation function and its particular case of image classification

2021 ◽  
Author(s):  
T.K. Biryukova
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Image Classification ◽  
Activation Function ◽  
Experimental Comparison ◽  
Training Time ◽  
Operation Speed ◽  
The Neural Network ◽  
Linear Algebraic Equations ◽  
Network Operation

Classic neural networks suppose trainable parameters to include just weights of neurons. This paper proposes parabolic integrodifferential splines (ID-splines), developed by author, as a new kind of activation function (AF) for neural networks, where ID-splines coefficients are also trainable parameters. Parameters of ID-spline AF together with weights of neurons are vary during the training in order to minimize the loss function thus reducing the training time and increasing the operation speed of the neural network. The newly developed algorithm enables software implementation of the ID-spline AF as a tool for neural networks construction, training and operation. It is proposed to use the same ID-spline AF for neurons in the same layer, but different for different layers. In this case, the parameters of the ID-spline AF for a particular layer change during the training process independently of the activation functions (AFs) of other network layers. In order to comply with the continuity condition for the derivative of the parabolic ID-spline on the interval (x x0, n) , its parameters fi (i= 0,...,n) should be calculated using the tridiagonal system of linear algebraic equations: To solve the system it is necessary to use two more equations arising from the boundary conditions for specific problems. For exam- ple the values of the grid function (if they are known) in the points (x x0, n) may be used for solving the system above: f f x0 = ( 0) , f f xn = ( n) . The parameters Iii+1 (i= 0,...,n−1 ) are used as trainable parameters of neural networks. The grid boundaries and spacing of the nodes of ID-spline AF are best chosen experimentally. The optimal selection of grid nodes allows improving the quality of results produced by the neural network. The formula for a parabolic ID-spline is such that the complexity of the calculations does not depend on whether the grid of nodes is uniform or non-uniform. An experimental comparison of the results of image classification from the popular FashionMNIST dataset by convolutional neural 0, x< 0 networks with the ID-spline AFs and the well-known ReLUx( ) =AF was carried out. The results reveal that the usage x x, ≥ 0 of the ID-spline AFs provides better accuracy of neural network operation than the ReLU AF. The training time for two convolutional layers network with two ID-spline AFs is just about 2 times longer than with two instances of ReLU AF. Doubling of the training time due to complexity of the ID-spline formula is the acceptable price for significantly better accuracy of the network. Wherein the difference of an operation speed of the networks with ID-spline and ReLU AFs will be negligible. The use of trainable ID-spline AFs makes it possible to simplify the architecture of neural networks without losing their efficiency. The modification of the well-known neural networks (ResNet etc.) by replacing traditional AFs with ID-spline AFs is a promising approach to increase the neural network operation accuracy. In a majority of cases, such a substitution does not require to train the network from scratch because it allows to use pre-trained on large datasets neuron weights supplied by standard software libraries for neural network construction thus substantially shortening training time.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Statistical Data ◽  
Activation Function ◽  
Decision Making Process ◽  
Neural Network Architecture ◽  
Acceptable Accuracy ◽  
The Neural Network ◽  
Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

scholarly journals Convolutional neural network for image classification based on transfer learning technique

2021 ◽  
Author(s):  
Ghassan Mohammed Halawani
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Image Classification ◽  
Transfer Learning ◽  
Learning Networks ◽  
The Neural Network ◽  
Learning Technique ◽  
Common Architecture

The main purpose of this project is to modify a convolutional neural network for image classification, based on a deep-learning framework. A transfer learning technique is used by the MATLAB interface to Alex-Net to train and modify the parameters in the last two fully connected layers of Alex-Net with a new dataset to perform classifications of thousands of images. First, the general common architecture of most neural networks and their benefits are presented. The mathematical models and the role of each part in the neural network are explained in detail. Second, different neural networks are studied in terms of architecture, application, and the working method to highlight the strengths and weaknesses of each of neural network. The final part conducts a detailed study on one of the most powerful deep-learning networks in image classification – i.e. the convolutional neural network – and how it can be modified to suit different classification tasks by using transfer learning technique in MATLAB.

Intelligent Classification and Measurement of Drill Wear

1994 ◽  
Vol 116 (3) ◽  
pp. 392-397 ◽  
Author(s):  
T. I. Liu ◽  
K. S. Anantharaman
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Flank Wear ◽  
Estimation Error ◽  
Back Propagation ◽  
Activation Function ◽  
The Neural Network ◽  
On Line ◽  
Thrust And Torque ◽  
Line Classification

Artificial neural networks are used for on-line classification and measurement of drill wear. The input vector of the neural network is obtained by processing the thrust and torque signals. Outputs are the wear states and flank wear measurements. The learning process can be performed by back propagation along with adaptive activation-function slope. The results of neural networks with and without adaptive activation-function slope, as well as various neural network architectures are compared. On-line classification of drill wear using neural networks has 100 percent reliability. The average flank wear estimation error using neural networks can be as low as 7.73 percent.

Exponential Stability of Periodic Solution for Impulsive Memristor-Based Cohen–Grossberg Neural Networks with Mixed Delays

2017 ◽  
Vol 31 (07) ◽  
pp. 1750022 ◽  
Author(s):  
Jiqiang Feng ◽  
Qiang Ma ◽  
Sitian Qin
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Periodic Solution ◽  
Exponential Stability ◽  
Sensor Arrays ◽  
Activation Function ◽  
Contraction Mapping Principle ◽  
Mixed Delays ◽  
The Neural Network ◽  
Inequality Technique

Memristor, as the future of artificial intelligence, has been widely used in pattern recognition or signal processing from sensor arrays. Memristor-based recurrent neural network (MRNN) is an ideal model to mimic the functionalities of the human brain due to the physical properties of memristor. In this paper, the periodicity for memristor-based Cohen–Grossberg neural networks (MCGNNs) is studied. The neural network (NN) considered in this paper is based on the memristor and involves time-varying delays, distributed delays and impulsive effects. The boundedness and monotonicity of the activation function are not assumed. By some inequality technique and contraction mapping principle, we prove the existence, uniqueness and exponential stability of periodic solution for MCGNNs. Finally, some numeral examples and comparisons are provided to illustrate the validation of our results.

scholarly journals Study of Convolutional Neural Networks applied to Image Stereo Matching

2020 ◽  
Author(s):  
João Pedro Poloni Ponce ◽  
Ricardo Suyama
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Error Rate ◽  
Stereo Matching ◽  
Structural Parameters ◽  
Neural Model ◽  
Temporal Difference ◽  
Training Time ◽  
The Neural Network

Stereo images are images formed from two or more sources that capture the same scene so that it is possible to infer the depth of the scene under analysis. The use of convolutional neural networks to compute these images has been shown to be a viable alternative due to its speed in finding the correspondence between the images. This raises questions related to the influence of structural parameters, such as size of kernel, stride and pooling policy on the performance of the neural network. To this end, this work sought to reproduce an article that deals with the topic and to explore the influence of the parameters mentioned above in function of the results of error rate and losses of the neural model. The results obtained reveal improvements. The influence of the parameters on the training time of the models was also notable, even using the GPU, the temporal difference in the training period between the maximum and minimum limits reached a ratio of six times.

Hyperbolic Hopfield neural networks for image classification in content-based image retrieval

2020 ◽  
pp. 2050059
Author(s):  
K. Anitha ◽  
R. Dhanalakshmi ◽  
K. Naresh ◽  
D. Rukmani Devi
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Image Retrieval ◽  
Image Classification ◽  
Retrieval System ◽  
Activation Function ◽  
Hopfield Neural Networks ◽  
Content Based Image Retrieval ◽  
Image Retrieval System ◽  
Complex Valued

Neural networks play a significant role in data classification. Complex-valued Hopfield Neural Network (CHNN) is mostly used in various fields including the image classification. Though CHNN has proven its credibility in the classification task, it has a few issues. Activation function of complex-valued neuron maps to a unit circle in the complex plane affecting the resolution factor, flexibility and compatibility to changes, during adaptation in retrieval systems. The proposed work demonstrates Content-Based Image Retrieval System (CBIR) with Hyperbolic Hopfield Neural Networks (HHNN), an analogue of CHNN for classifying images. Activation function of the Hyperbolic neuron is not cyclic in hyperbolic plane. The images are mathematically represented and indexed using the six basic features. The proposed HHNN classifier is trained, tested and evaluated through extensive experiments considering individual features and four combined features for indexing. The obtained results prove that HHNN guides retrieval process, enhances system performance and minimizes the cost of implementing Neural Network Classifier-based image retrieval system.

SYNTHESIS OF NEURAL NETWORK BASED ON PETRY NET USING AS EXAMPLE THE PROBLEM OF MOVEMENT AND STABILIZATION OF STRUCTURE GROUPS OF UNMANNED AERIAL VEHICLES

2018 ◽  
pp. 26-33
Author(s):  
Dmitry Olegovich Romannikov ◽  
Alexander Aleksandrovich Voevoda
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Petri Nets ◽  
Unmanned Aerial Vehicles ◽  
Training Time ◽  
Initial Values ◽  
Network Training ◽  
Aerial Vehicles ◽  
The Neural Network ◽  
Parameter Values

The article focuses on the approach to forming the structure of a neural network with application of a pre-built algorithm using Petri nets which represent a well-characterized body of mathematics and help to describe algorithms, in particular, distributed asynchronous systems. According to the proposed approach, the model built in Petri nets serves as the basis for further developing the neural network. There was proposed the idea of informal transformation, which makes sense because the structure of Petri net provides substantiation for the structure of the neural network. This fact leads to decreasing the number of training parameters in the neural network (in the example described in the article the decrease was more than twice: from 650 to 254), increasing the time of the network training and getting initial values for the training parameters. It has been stated that with the initial values obtained the training time grows even more and, thus, training process acts as fine-adjusting values of parameters. Transformation can be explained by the fact that both Petri nets and neural networks act as languages for describing functions, and differ only in the case of neural networks, where the presented function must be trained first (or to find parameter values). The above-mentioned approach is illustrated by the example of the problem of automatic formation of a group of unmanned aerial vehicles (UAV) and their movement. In this problem, identical instances of the neural network are located on each UAV and interact in asynchronous mode.

scholarly journals Obtaining Deep Learning Models for Automatic Classification of Leukocytes

2020 ◽  
pp. 1-32
Author(s):  
Pedro João Rodrigues ◽  
Getúlio Peixoto Igrejas ◽  
Romeu Ferreira Beato
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Image Classification ◽  
Network Architectures ◽  
Learning Models ◽  
Training Time ◽  
The Neural Network ◽  
Regularization Techniques ◽  
Microscopy Images

In this work, the authors classify leukocyte images using the neural network architectures that won the annual ILSVRC competition. The classification of leukocytes is made using pretrained networks and the same networks trained from scratch in order to select the ones that achieve the best performance for the intended task. The categories used are eosinophils, lymphocytes, monocytes, and neutrophils. The analysis of the results takes into account the amount of training required, the regularization techniques used, the training time, and the accuracy in image classification. The best classification results, on the order of 98%, suggest that it is possible, considering a competent preprocessing, to train a network like the DenseNet with 169 or 201 layers, in about 100 epochs, to classify leukocytes in microscopy images.

scholarly journals Evaluating the Impact of Optical Interconnects on a Multi-Chip Machine-Learning Architecture

Electronics ◽  
2018 ◽  
Vol 7 (8) ◽  
pp. 130 ◽  
Author(s):  
Yuhwan Ro ◽  
Eojin Lee ◽  
Jung Ahn
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Optical Interconnects ◽  
Performance Model ◽  
Training Time ◽  
Performance Improvements ◽  
Cluster Architecture ◽  
The Neural Network ◽  
The Impact

Following trends that emphasize neural networks for machine learning, many studies regarding computing systems have focused on accelerating deep neural networks. These studies often propose utilizing the accelerator specialized in a neural network and the cluster architecture composed of interconnected accelerator chips. We observed that inter-accelerator communication within a cluster has a significant impact on the training time of the neural network. In this paper, we show the advantages of optical interconnects for multi-chip machine-learning architecture by demonstrating performance improvements through replacing electrical interconnects with optical ones in an existing multi-chip system. We propose to use highly practical optical interconnect implementation and devise an arithmetic performance model to fairly assess the impact of optical interconnects on a machine-learning accelerator platform. In our evaluation of nine Convolutional Neural Networks with various input sizes, 100 and 400 Gbps optical interconnects reduce the training time by an average of 20.6% and 35.6%, respectively, compared to the baseline system with 25.6 Gbps electrical ones.

scholarly journals Convolutional neural network for image classification based on transfer learning technique

2021 ◽  
Author(s):  
Ghassan Mohammed Halawani
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Image Classification ◽  
Transfer Learning ◽  
Learning Networks ◽  
The Neural Network ◽  
Learning Technique ◽  
Common Architecture

The main purpose of this project is to modify a convolutional neural network for image classification, based on a deep-learning framework. A transfer learning technique is used by the MATLAB interface to Alex-Net to train and modify the parameters in the last two fully connected layers of Alex-Net with a new dataset to perform classifications of thousands of images. First, the general common architecture of most neural networks and their benefits are presented. The mathematical models and the role of each part in the neural network are explained in detail. Second, different neural networks are studied in terms of architecture, application, and the working method to highlight the strengths and weaknesses of each of neural network. The final part conducts a detailed study on one of the most powerful deep-learning networks in image classification – i.e. the convolutional neural network – and how it can be modified to suit different classification tasks by using transfer learning technique in MATLAB.

Sign in / Sign up

Export Citation Format

Share Document