scholarly journals Coarse-Grained Pruning of Neural Network Models Based on Blocky Sparse Structure

Entropy ◽  
2021 ◽  
Vol 23 (8) ◽  
pp. 1042
Author(s):  
Lan Huang ◽  
Jia Zeng ◽  
Shiqi Sun ◽  
Wencong Wang ◽  
Yan Wang ◽  
...  
Keyword(s):  
Neural Network ◽  
Sparse Matrix ◽  
Computational Cost ◽  
Network Models ◽  
Coarse Grained ◽  
Model Parameters ◽  
Neural Network Models ◽  
Convolutional Networks ◽  
Research Fields ◽  
Pruning Strategy

Deep neural networks may achieve excellent performance in many research fields. However, many deep neural network models are over-parameterized. The computation of weight matrices often consumes a lot of time, which requires plenty of computing resources. In order to solve these problems, a novel block-based division method and a special coarse-grained block pruning strategy are proposed in this paper to simplify and compress the fully connected structure, and the pruned weight matrices with a blocky structure are then stored in the format of Block Sparse Row (BSR) to accelerate the calculation of the weight matrices. First, the weight matrices are divided into square sub-blocks based on spatial aggregation. Second, a coarse-grained block pruning procedure is utilized to scale down the model parameters. Finally, the BSR storage format, which is much more friendly to block sparse matrix storage and computation, is employed to store these pruned dense weight blocks to speed up the calculation. In the following experiments on MNIST and Fashion-MNIST datasets, the trend of accuracies with different pruning granularities and different sparsity is explored in order to analyze our method. The experimental results show that our coarse-grained block pruning method can compress the network and can reduce the computational cost without greatly degrading the classification accuracy. The experiment on the CIFAR-10 dataset shows that our block pruning strategy can combine well with the convolutional networks.

scholarly journals A Primer on Neural Network Models for Natural Language Processing

2016 ◽  
Vol 57 ◽  
pp. 345-420 ◽  
Author(s):  
Yoav Goldberg
Keyword(s):  
Neural Network ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Speech Processing ◽  
Network Models ◽  
Neural Network Models ◽  
Convolutional Networks ◽  
The Past ◽  
Gradient Computation

Over the past few years, neural networks have re-emerged as powerful machine-learning models, yielding state-of-the-art results in fields such as image recognition and speech processing. More recently, neural network models started to be applied also to textual natural language signals, again with very promising results. This tutorial surveys neural network models from the perspective of natural language processing research, in an attempt to bring natural-language researchers up to speed with the neural techniques. The tutorial covers input encoding for natural language tasks, feed-forward networks, convolutional networks, recurrent networks and recursive networks, as well as the computation graph abstraction for automatic gradient computation.

scholarly journals Evaluating Representations for Gene Ontology Terms

2019 ◽  
Author(s):  
Dat Duong ◽  
Ankith Uppunda ◽  
Lisa Gai ◽  
Chelsea Ju ◽  
James Zhang ◽  
...  
Keyword(s):  
Neural Network ◽  
Gene Ontology ◽  
Network Models ◽  
Language Models ◽  
Future Research ◽  
Model Parameters ◽  
Neural Network Models ◽  
Go Annotation ◽  
Go Terms ◽  
Similarity Scores

AbstractProtein functions can be described by the Gene Ontology (GO) terms, allowing us to compare the functions of two proteins by measuring the similarity of the terms assigned to them. Recent works have applied neural network models to derive the vector representations for GO terms and compute similarity scores for these terms by comparing their vector embeddings. There are two typical ways to embed GO terms into vectors; a model can either embed the definitions of the terms or the topology of the terms in the ontology. In this paper, we design three tasks to critically evaluate the GO embeddings of two recent neural network models, and further introduce additional models for embedding GO terms, adapted from three popular neural network frameworks: Graph Convolution Network (GCN), Embeddings from Language Models (ELMo), and Bidirectional Encoder Representations from Transformers (BERT), which have not yet been explored in previous works. Task 1 studies edge cases where the GO embeddings may not provide meaningful similarity scores for GO terms. We find that all neural network based methods fail to produce high similarity scores for related terms when these terms have low Information Content values. Task 2 is a canonical task which estimates how well GO embeddings can compare functions of two orthologous genes or two interacting proteins. The best neural network methods for this task are those that embed GO terms using their definitions, and the differences among such methods are small. Task 3 evaluates how GO embeddings affect the performance of GO annotation methods, which predict whether a protein should be labeled by certain GO terms. When the annotation datasets contain many samples for each GO label, GO embeddings do not improve the classification accuracy. Machine learning GO annotation methods often remove rare GO labels from the training datasets so that the model parameters can be efficiently trained. We evaluate whether GO embeddings can improve prediction of rare labels unseen in the training datasets, and find that GO embeddings based on the BERT framework achieve the best results in this setting. We present our embedding methods and three evaluation tasks as the basis for future research on this topic.

scholarly journals Validation of Text Data Preprocessing Using a Neural Network Model

2020 ◽  
Vol 2020 ◽  
pp. 1-9
Author(s):  
HoSung Woo ◽  
JaMee Kim ◽  
WonGyu Lee
Keyword(s):  
Neural Network ◽  
Network Model ◽  
Neural Network Model ◽  
Text Processing ◽  
Network Models ◽  
Data Preprocessing ◽  
Neural Network Models ◽  
Exploration Process ◽  
Positive Effects ◽  
Research Fields

Many artificial intelligence studies focus on designing new neural network models or optimizing hyperparameters to improve model accuracy. To develop a reliable model, appropriate data are required, and data preprocessing is an essential part of acquiring the data. Although various studies regard data preprocessing as part of the data exploration process, those studies lack awareness about the need for separate technologies and solutions for preprocessing. Therefore, this study evaluated combinations of preprocessing types in a text-processing neural network model. Better performance was observed when two preprocessing types were used than when three or more preprocessing types were used for data purification. More specifically, using lemmatization and punctuation splitting together, lemmatization and lowering together, and lowering and punctuation splitting together showed positive effects on accuracy. This study is significant because the results allow better decisions to be made about the selection of the preprocessing types in various research fields, including neural network research.

scholarly journals A Novel Low-Bit Quantization Strategy for Compressing Deep Neural Networks

2020 ◽  
Vol 2020 ◽  
pp. 1-7
Author(s):  
Xin Long ◽  
XiangRong Zeng ◽  
Zongcheng Ben ◽  
Dianle Zhou ◽  
Maojun Zhang
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Computational Cost ◽  
Network Models ◽  
Memory Consumption ◽  
Neural Network Models ◽  
The Neural Network ◽  
The Neural Networks ◽  
Novel Strategy

The increase in sophistication of neural network models in recent years has exponentially expanded memory consumption and computational cost, thereby hindering their applications on ASIC, FPGA, and other mobile devices. Therefore, compressing and accelerating the neural networks are necessary. In this study, we introduce a novel strategy to train low-bit networks with weights and activations quantized by several bits and address two corresponding fundamental issues. One is to approximate activations through low-bit discretization for decreasing network computational cost and dot-product memory. The other is to specify weight quantization and update mechanism for discrete weights to avoid gradient mismatch. With quantized low-bit weights and activations, the costly full-precision operation will be replaced by shift operation. We evaluate the proposed method on common datasets, and results show that this method can dramatically compress the neural network with slight accuracy loss.

scholarly journals Uncertainpy: A Python toolbox for uncertainty quantification and sensitivity analysis in computational neuroscience

2018 ◽  
Author(s):  
Simen Tennøe ◽  
Geir Halnes ◽  
Gaute T. Einevoll
Keyword(s):  
Neural Network ◽  
Sensitivity Analysis ◽  
Action Potential ◽  
Uncertainty Quantification ◽  
Open Source ◽  
Computational Neuroscience ◽  
Network Models ◽  
Model Parameters ◽  
Model Output ◽  
Neural Network Models

AbstractComputational models in neuroscience typically contain many parameters that are poorly constrained by experimental data. Uncertainty quantification and sensitivity analysis provide rigorous procedures to quantify how the model output depends on this parameter uncertainty. Unfortunately, the application of such methods is not yet standard within the field of neuroscience.Here we present Uncertainpy, an open-source Python toolbox, tailored to perform uncertainty quantification and sensitivity analysis of neuroscience models. Uncertainpy aims to make it easy and quick to get started with uncertainty analysis, without any need for detailed prior knowledge. The toolbox allows uncertainty quantification and sensitivity analysis to be performed on already existing models without needing to modify the model equations or model implementation. Uncertainpy bases its analysis on polynomial chaos expansions, which are more efficient than the more standard Monte-Carlo based approaches.Uncertainpy is tailored for neuroscience applications by its built-in capability for calculating characteristic features in the model output. The toolbox does not merely perform a point-to- point comparison of the “raw” model output (e.g. membrane voltage traces), but can also calculate the uncertainty and sensitivity of salient model response features such as spike timing, action potential width, mean interspike interval, and other features relevant for various neural and neural network models. Uncertainpy comes with several common models and features built in, and including custom models and new features is easy.The aim of the current paper is to present Uncertainpy for the neuroscience community in a user- oriented manner. To demonstrate its broad applicability, we perform an uncertainty quantification and sensitivity analysis on three case studies relevant for neuroscience: the original Hodgkin-Huxley point-neuron model for action potential generation, a multi-compartmental model of a thalamic interneuron implemented in the NEURON simulator, and a sparsely connected recurrent network model implemented in the NEST simulator.SIGNIFICANCE STATEMENTA major challenge in computational neuroscience is to specify the often large number of parameters that define the neuron and neural network models. Many of these parameters have an inherent variability, and some may even be actively regulated and change with time. It is important to know how the uncertainty in model parameters affects the model predictions. To address this need we here present Uncertainpy, an open-source Python toolbox tailored to perform uncertainty quantification and sensitivity analysis of neuroscience models.

Computational cost improvement of neural network models in black box nonlinear system identification

2015 ◽  
Vol 166 ◽  
pp. 96-108 ◽  
Author(s):  
Hector M. Romero Ugalde ◽  
Jean-Claude Carmona ◽  
Juan Reyes-Reyes ◽  
Victor M. Alvarado ◽  
Juan Mantilla
Keyword(s):  
Neural Network ◽  
System Identification ◽  
Nonlinear System ◽  
Nonlinear System Identification ◽  
Computational Cost ◽  
Network Models ◽  
Black Box ◽  
Neural Network Models

scholarly journals Multipurpose Watermarking Approach for Copyright and Integrity of Steganographic Autoencoder Models

2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Wei Gu ◽  
Ching-Chun Chang ◽  
Yu Bai ◽  
Yunyuan Fan ◽  
Liang Tao ◽  
...  
Keyword(s):  
Neural Network ◽  
Network Models ◽  
Model Parameters ◽  
Learning Technology ◽  
Neural Network Models ◽  
Modified Model ◽  
Copyright Information ◽  
New Type ◽  
Multipurpose Watermarking ◽  
Computational Resources

With the great achievements of deep learning technology, neural network models have emerged as a new type of intellectual property. Neural network models’ design and training require considerable computational resources and time. Watermarking is a potential solution for achieving copyright protection and integrity of neural network models without excessively compromising the models’ accuracy and stability. In this work, we develop a multipurpose watermarking method for securing the copyright and integrity of a steganographic autoencoder referred to as “HiDDen.” This autoencoder model is used to hide different kinds of watermark messages in digital images. Copyright information is embedded with imperceptibly modified model parameters, and integrity is verified by embedding the Hash value generated from the model parameters. Experimental results show that the proposed multipurpose watermarking method can reliably identify copyright ownership and localize tampered parts of the model parameters. Furthermore, the accuracy and robustness of the autoencoder model are perfectly preserved.

Forecasting the Energy Consumption of an Industrial Enterprise Based on the Neural Network Model

2021 ◽  
Vol 23 ◽  
pp. 484-492
Author(s):  
Vasyl Kalinchyk ◽  
Olexandr Meita ◽  
Vitalii Pobigaylo ◽  
Vitalii Kalinchyk ◽  
Danylo Filyanin
Keyword(s):  
Neural Network ◽  
Energy Consumption ◽  
Comparative Analysis ◽  
Neural Network Model ◽  
Network Models ◽  
Model Parameters ◽  
Training Algorithms ◽  
Neural Network Models ◽  
The Neural Network ◽  
And Training

This research paper investigates the application of neural network models for forecasting in energy. The results of forecasting the weekly energy consumption of the enterprise according to the model of a multilayer perceptron at different values of neurons and training algorithms are given. The estimation and comparative analysis of models depending on model parameters is made.

Method of complex copper-zinc ore typification using neural network models

2020 ◽  
Vol 5 ◽  
pp. 140-147 ◽  
Author(s):  
T.N. Aleksandrova ◽  
◽  
E.K. Ushakov ◽  
A.V. Orlova ◽  
◽  
...  
Keyword(s):  
Neural Network ◽  
Network Models ◽  
Neural Network Models ◽  
Copper Zinc ◽  
Complex Copper

Digital twin of equipment as a basis for the consumer in digital production

2020 ◽  
Keyword(s):  
Neural Network ◽  
Tool Wear ◽  
Chip Formation ◽  
Network Models ◽  
Machining Accuracy ◽  
Neural Network Models ◽  
Digital Twin ◽  
The Neural Network ◽  
Digital Production ◽  
Cyberphysical System

The neural network models series used in the development of an aggregated digital twin of equipment as a cyber-physical system are presented. The twins of machining accuracy, chip formation and tool wear are examined in detail. On their basis, systems for stabilization of the chip formation process during cutting and diagnose of the cutting too wear are developed. Keywords cyberphysical system; neural network model of equipment; big data, digital twin of the chip formation; digital twin of the tool wear; digital twin of nanostructured coating choice

Sign in / Sign up

Export Citation Format

Share Document