scholarly journals Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?

2020 ◽  
Author(s):  
Lifu Wang ◽  
Bo Shen ◽  
Ning Zhao ◽  
Zhiyuan Zhang
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Level Sets ◽  
Theoretical Explanation ◽  
Learning Ability ◽  
Local Minima ◽  
Global Minima ◽  
Residual Network ◽  
The Neural Network ◽  
Hidden Nodes

The residual network is now one of the most effective structures in deep learning, which utilizes the skip connections to “guarantee" the performance will not get worse. However, the non-convexity of the neural network makes it unclear whether the skip connections do provably improve the learning ability since the nonlinearity may create many local minima. In some previous works [Freeman and Bruna, 2016], it is shown that despite the non-convexity, the loss landscape of the two-layer ReLU network has good properties when the number m of hidden nodes is very large. In this paper, we follow this line to study the topology (sub-level sets) of the loss landscape of deep ReLU neural networks with a skip connection and theoretically prove that the skip connection network inherits the good properties of the two-layer network and skip connections can help to control the connectedness of the sub-level sets, such that any local minima worse than the global minima of some two-layer ReLU network will be very “shallow". The “depth" of these local minima are at most O(m^(η-1)/n), where n is the input dimension, η<1. This provides a theoretical explanation for the effectiveness of the skip connection in deep learning.

scholarly journals Increasing of Thermal Images Resolution Using Deep Learning Neural Networks

2021 ◽  
Vol 25 (3) ◽  
pp. 31-35
Author(s):  
Piotr Więcek ◽  
Dominik Sankowski
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Execution Time ◽  
High Accuracy ◽  
New Method ◽  
Residual Network ◽  
Thermal Images ◽  
The Neural Network

The article presents a new algorithm for increasing the resolution of thermal images. For this purpose, the residual network was integrated with the Kernel-Sharing Atrous Convolution (KSAC) image sub-sampling module. A significant reduction in the algorithm’s complexity and shortening the execution time while maintaining high accuracy were achieved. The neural network has been implemented in the PyTorch environment. The results of the proposed new method of increasing the resolution of thermal images with sizes 32 × 24, 160 × 120 and 640 × 480 for scales up to 6 are presented.

scholarly journals Augmented Reality Maintenance Assistant Using YOLOv5

2021 ◽  
Vol 11 (11) ◽  
pp. 4758
Author(s):  
Ana Malta ◽  
Mateus Mendes ◽  
Torres Farinha
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Object Recognition ◽  
Augmented Reality ◽  
Real Time ◽  
Recognition System ◽  
High Accuracy ◽  
Video Streams ◽  
The Neural Network ◽  
Deep Learning Neural Network

Maintenance professionals and other technical staff regularly need to learn to identify new parts in car engines and other equipment. The present work proposes a model of a task assistant based on a deep learning neural network. A YOLOv5 network is used for recognizing some of the constituent parts of an automobile. A dataset of car engine images was created and eight car parts were marked in the images. Then, the neural network was trained to detect each part. The results show that YOLOv5s is able to successfully detect the parts in real time video streams, with high accuracy, thus being useful as an aid to train professionals learning to deal with new equipment using augmented reality. The architecture of an object recognition system using augmented reality glasses is also designed.

The Optimization of Adaptive PID Control Algorithm Based on RBF Neural Network

2014 ◽  
Vol 998-999 ◽  
pp. 943-946
Author(s):  
Jing Liu ◽  
Guo Xin Wang
Keyword(s):  
Neural Network ◽  
Pid Controller ◽  
Rbf Neural Network ◽  
High Reliability ◽  
Simple Algorithm ◽  
Poor Performance ◽  
Learning Ability ◽  
The Neural Network ◽  
Pid Controller Tuning ◽  
Deterministic Control

As the earliest practical controller, PID controller has more than 50 years of history, and it is still the most widely used and most common industrial controllers. PID controller is simple to understand and use, without a prerequisite for an accurate model of the physical system, thus become the most popular, the most common controller. The reason why PID controller is the first developed one is that its simple algorithm, robustness and high reliability. It is widely used in process control and motion control, especially for accurate mathematical model that can be established deterministic control system. But the conventional PID controller tuning parameters are often poor performance, poor adaptability to the operating environment. The neural network has a strong nonlinear mapping ability, competence, self-learning ability of associative memory, and has a viable quantities of information processing methods and good fault tolerance.

GTAW Procedure Expert System Based on Neural Network

2013 ◽  
Vol 455 ◽  
pp. 425-430 ◽  
Author(s):  
Xue Wu Wang ◽  
Shang Yong Yang
Keyword(s):  
Neural Network ◽  
Expert System ◽  
Expert Knowledge ◽  
Welding Process ◽  
Learning Ability ◽  
Welding Parameters ◽  
Neural Network Learning ◽  
The Neural Network ◽  
Function Design ◽  
Welding Procedure

Intelligent procedure expert system was developed to select appropriate GTAW procedure in this paper. First, the function design and implementation methods of the welding procedure expert system were introduced. The expert system can present the welding procedure card, multimedia display of welding process, and output function to makes the data sharing more convenient. Then, the database design of the welding procedure expert system based on C/S mode was presented where the expert knowledge was stored. At last, the neural network model was established to realize procedure selection based on the neural network learning ability and the welding case from the database. With the BPNN model, the welding parameters can be obtained based on the input welding conditions.

scholarly journals Hyperparameters optimization for ResNet and Xception in the purpose of diagnosing COVID-19

2021 ◽  
pp. 1-17
Author(s):  
Hania H. Farag ◽  
Lamiaa A. A. Said ◽  
Mohamed R. M. Rizk ◽  
Magdy Abd ElAzim Ahmed
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Network ◽  
Random Search ◽  
Learning Networks ◽  
Residual Network ◽  
Global Pandemic ◽  
Search Optimization

COVID-19 has been considered as a global pandemic. Recently, researchers are using deep learning networks for medical diseases’ diagnosis. Some of these researches focuses on optimizing deep learning neural networks for enhancing the network accuracy. Optimizing the Convolutional Neural Network includes testing various networks which are obtained through manually configuring their hyperparameters, then the configuration with the highest accuracy is implemented. Each time a different database is used, a different combination of the hyperparameters is required. This paper introduces two COVID-19 diagnosing systems using both Residual Network and Xception Network optimized by random search in the purpose of finding optimal models that give better diagnosis rates for COVID-19. The proposed systems showed that hyperparameters tuning for the ResNet and the Xception Net using random search optimization give more accurate results than other techniques with accuracies 99.27536% and 100 % respectively. We can conclude that hyperparameters tuning using random search optimization for either the tuned Residual Network or the tuned Xception Network gives better accuracies than other techniques diagnosing COVID-19.

scholarly journals Neutrosophy for physiological data compression: in particular by neural nets using deep learning

2020 ◽  
pp. 74-80
Author(s):  
Philippe Schweizer ◽  
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Boundary Surface ◽  
Human Perception ◽  
Neural Nets ◽  
Physiological Data ◽  
Compression Process ◽  
The Neural Network ◽  
Algorithmic Solution ◽  
Algorithmic Techniques

We would like to show the small distance in neutropsophy applications in sciences and humanities, has both finally consider as a terminal user a human. The pace of data production continues to grow, leading to increased needs for efficient storage and transmission. Indeed, the consumption of this information is preferably made on mobile terminals using connections invoiced to the user and having only reduced storage capacities. Deep learning neural networks have recently exceeded the compression rates of algorithmic techniques for text. We believe that they can also significantly challenge classical methods for both audio and visual data (images and videos). To obtain the best physiological compression, i.e. the highest compression ratio because it comes closest to the specificity of human perception, we propose using a neutrosophical representation of the information for the entire compression-decompression cycle. Such a representation consists for each elementary information to add to it a simple neutrosophical number which informs the neural network about its characteristics relative to compression during this treatment. Such a neutrosophical number is in fact a triplet (t,i,f) representing here the belonging of the element to the three constituent components of information in compression; 1° t = the true significant part to be preserved, 2° i = the inderterminated redundant part or noise to be eliminated in compression and 3° f = the false artifacts being produced in the compression process (to be compensated). The complexity of human perception and the subtle niches of its defects that one seeks to exploit requires a detailed and complex mapping that a neural network can produce better than any other algorithmic solution, and networks with deep learning have proven their ability to produce a detailed boundary surface in classifiers.

scholarly journals Sub3DNet1.0: a deep-learning model for regional-scale 3D subsurface structure mapping

2021 ◽  
Vol 14 (6) ◽  
pp. 3421-3435
Author(s):  
Zhenjiao Jiang ◽  
Dirk Mallants ◽  
Lei Gao ◽  
Tim Munday ◽  
Gregoire Mariethoz ◽  
...  
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Land Surface ◽  
Regional Scale ◽  
Learning Model ◽  
Generic Structure ◽  
Squared Error ◽  
The Neural Network ◽  
Deep Learning Model ◽  
Surface Observations

Abstract. This study introduces an efficient deep-learning model based on convolutional neural networks with joint autoencoder and adversarial structures for 3D subsurface mapping from 2D surface observations. The method was applied to delineate paleovalleys in an Australian desert landscape. The neural network was trained on a 6400 km2 domain by using a land surface topography as 2D input and an airborne electromagnetic (AEM)-derived probability map of paleovalley presence as 3D output. The trained neural network has a squared error <0.10 across 99 % of the training domain and produces a squared error <0.10 across 93 % of the validation domain, demonstrating that it is reliable in reconstructing 3D paleovalley patterns beyond the training area. Due to its generic structure, the neural network structure designed in this study and the training algorithm have broad application potential to construct 3D geological features (e.g., ore bodies, aquifer) from 2D land surface observations.

Pattern-Based Deep Learning Method to Extract Information from the Log Dataset

2021 ◽  
pp. 2150296
Author(s):  
Xi Li ◽  
Ting Wang ◽  
Shexiong Wang
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Data Sets ◽  
Learning Method ◽  
Statistical Features ◽  
Log Data ◽  
The Neural Network ◽  
Testing Data ◽  
Extract Information

It draws researchers’ attentions how to make use of the log data effectively without paying much for storing them. In this paper, we propose pattern-based deep learning method to extract the features from log datasets and to facilitate its further use at the reasonable expense of the storage performances. By taking the advantages of the neural network and thoughts to combine statistical features with experts’ knowledge, there are satisfactory results in the experiments on some specified datasets and on the routine systems that our group maintains. Processed on testing data sets, the model is 5%, at least, more likely to outperform its competitors in accuracy perspective. More importantly, its schema unveils a new way to mingle experts’ experiences with statistical log parser.

Cluster Based Medical Image Registration Using Optimized Neural Network

2017 ◽  
pp. 1437-1467
Author(s):  
Joydev Hazra ◽  
Aditi Roy Chowdhury ◽  
Paramartha Dutta
Keyword(s):  
Neural Network ◽  
Genetic Algorithm ◽  
Image Registration ◽  
Convergence Rate ◽  
Optimization Algorithm ◽  
Clustering Algorithms ◽  
Data Sets ◽  
Learning Method ◽  
Local Minima ◽  
The Neural Network

Registration of medical images like CT-MR, MR-MR etc. are challenging area for researchers. This chapter introduces a new cluster based registration technique with help of the supervised optimized neural network. Features are extracted from different cluster of an image obtained from clustering algorithms. To overcome the drawback regarding convergence rate of neural network, an optimized neural network is proposed in this chapter. The weights are optimized to increase the convergence rate as well as to avoid stuck in local minima. Different clustering algorithms are explored to minimize the clustering error of an image and extract features from suitable one. The supervised learning method applied to train the neural network. During this training process an optimization algorithm named Genetic Algorithm (GA) is used to update the weights of a neural network. To demonstrate the effectiveness of the proposed method, investigation is carried out on MR T1, T2 data sets. The proposed method shows convincing results in comparison with other existing techniques.

Sign in / Sign up

Export Citation Format

Share Document