Deep neural networks for multimodal data fusion and affect recognition

2020 ◽  
Vol 7 (2) ◽  
pp. 130
Author(s):  
Dhruv Bhandari ◽  
Sandeep Paul ◽  
Apurva Narayan
Keyword(s):  
Neural Networks ◽  
Data Fusion ◽  
Deep Neural Networks ◽  
Affect Recognition ◽  
Multimodal Data ◽  
Multimodal Data Fusion

Multimodal data fusion framework enhanced robot-assisted minimally invasive surgery

2021 ◽  
pp. 014233122098435
Author(s):  
Wen Qi ◽  
Hang Su ◽  
Ke Fan ◽  
Ziyang Chen ◽  
Jiehao Li ◽  
...  
Keyword(s):  
Minimally Invasive Surgery ◽  
Minimally Invasive ◽  
Data Fusion ◽  
Invasive Surgery ◽  
Robot Control ◽  
Surgical Robot ◽  
Robot Assisted ◽  
Multimodal Data ◽  
Multimodal Data Fusion ◽  
Fusion Framework

The generous application of robot-assisted minimally invasive surgery (RAMIS) promotes human-machine interaction (HMI). Identifying various behaviors of doctors can enhance the RAMIS procedure for the redundant robot. It bridges intelligent robot control and activity recognition strategies in the operating room, including hand gestures and human activities. In this paper, to enhance identification in a dynamic situation, we propose a multimodal data fusion framework to provide multiple information for accuracy enhancement. Firstly, a multi-sensors based hardware structure is designed to capture varied data from various devices, including depth camera and smartphone. Furthermore, in different surgical tasks, the robot control mechanism can shift automatically. The experimental results evaluate the efficiency of developing the multimodal framework for RAMIS by comparing it with a single sensor system. Implementing the KUKA LWR4+ in a surgical robot environment indicates that the surgical robot systems can work with medical staff in the future.

scholarly journals Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation

2020 ◽  
Vol 64 ◽  
pp. 149-187 ◽  
Author(s):  
Yu-Dong Zhang ◽  
Zhengchao Dong ◽  
Shui-Hua Wang ◽  
Xiang Yu ◽  
Xujing Yao ◽  
...  
Keyword(s):  
Data Fusion ◽  
Multimodal Data ◽  
Multimodal Data Fusion

scholarly journals Exploring Multimodal Data Fusion Through Joint Decompositions with Flexible Couplings

2016 ◽  
Vol 64 (18) ◽  
pp. 4830-4844 ◽  
Author(s):  
Rodrigo Cabral Farias ◽  
Jeremy Emile Cohen ◽  
Pierre Comon
Keyword(s):  
Data Fusion ◽  
Multimodal Data ◽  
Multimodal Data Fusion

Damage Detection of Composite Materials Using Data Fusion With Deep Neural Networks

2020 ◽  
Author(s):  
Shweta Dabetwar ◽  
Stephen Ekwaro-Osire ◽  
João Paulo Dias
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Composite Materials ◽  
Data Fusion ◽  
Damage Detection ◽  
Convolutional Neural Network ◽  
Deep Neural Networks ◽  
Mixed Data ◽  
Image Encoding ◽  
Using Data

Abstract Composite materials have enormous applications in various fields. Thus, it is important to have an efficient damage detection method to avoid catastrophic failures. Due to the existence of multiple damage modes and the availability of data in different formats, it is important to employ efficient techniques to consider all the types of damage. Deep neural networks were seen to exhibit the ability to address similar complex problems. The research question in this work is ‘Can data fusion improve damage classification using the convolutional neural network?’ The specific aims developed were to 1) assess the performance of image encoding algorithms, 2) classify the damage using data from separate experimental coupons, and 3) classify the damage using mixed data from multiple experimental coupons. Two different experimental measurements were taken from NASA Ames Prognostic Repository for Carbon Fiber Reinforced polymer. To use data fusion, the piezoelectric signals were converted into images using Gramian Angular Field (GAF) and Markov Transition Field. Using data fusion techniques, the input dataset was created for a convolutional neural network with three hidden layers to determine the damage states. The accuracies of all the image encoding algorithms were compared. The analysis showed that data fusion provided better results as it contained more information on the damages modes that occur in composite materials. Additionally, GAF was shown to perform the best. Thus, the combination of data fusion and deep neural network techniques provides an efficient method for damage detection of composite materials.

Sign in / Sign up

Export Citation Format

Share Document