Semantic-Based Sentence Recognition in Images Using Bimodal Deep Learning

AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

Nature Communications ◽

10.1038/s41467-021-25637-w ◽

2021 ◽

Vol 12 (1) ◽

Cited By ~ 2

Author(s):

Feng Wen ◽

Zixuan Zhang ◽

Tianyiyi He ◽

Chengkuo Lee

Keyword(s):

Deep Learning ◽

Sign Language ◽

Learning Model ◽

Virtual Space ◽

New Order ◽

Language Recognition ◽

Sign Language Recognition ◽

Sentence Recognition ◽

Bidirectional Communication ◽

Deep Learning Model

AbstractSign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers.

Download Full-text

Effects of Fundamental Frequency Contours on Sentence Recognition in Mandarin-Speaking Children With Cochlear Implants

Journal of Speech Language and Hearing Research ◽

10.1044/2020_jslhr-20-00033 ◽

2020 ◽

Vol 63 (11) ◽

pp. 3855-3864

Author(s):

Wanting Huang ◽

Lena L. N. Wong ◽

Fei Chen ◽

Haihong Liu ◽

Wei Liang

Keyword(s):

Cochlear Implants ◽

Fundamental Frequency ◽

Signal To Noise Ratio ◽

Lexical Tone ◽

Signal To Noise ◽

Sentence Recognition ◽

Test Conditions ◽

Age Appropriate ◽

F0 Contour ◽

Appropriate Sentences

Purpose Fundamental frequency (F0) is the primary acoustic cue for lexical tone perception in tonal languages but is processed in a limited way in cochlear implant (CI) systems. The aim of this study was to evaluate the importance of F0 contours in sentence recognition in Mandarin-speaking children with CIs and find out whether it is similar to/different from that in age-matched normal-hearing (NH) peers. Method Age-appropriate sentences, with F0 contours manipulated to be either natural or flattened, were randomly presented to preschool children with CIs and their age-matched peers with NH under three test conditions: in quiet, in white noise, and with competing sentences at 0 dB signal-to-noise ratio. Results The neutralization of F0 contours resulted in a significant reduction in sentence recognition. While this was seen only in noise conditions among NH children, it was observed throughout all test conditions among children with CIs. Moreover, the F0 contour-induced accuracy reduction ratios (i.e., the reduction in sentence recognition resulting from the neutralization of F0 contours compared to the normal F0 condition) were significantly greater in children with CIs than in NH children in all test conditions. Conclusions F0 contours play a major role in sentence recognition in both quiet and noise among pediatric implantees, and the contribution of the F0 contour is even more salient than that in age-matched NH children. These results also suggest that there may be differences between children with CIs and NH children in how F0 contours are processed.

Download Full-text

Deep Learning

10.1017/cbo9780511780295 ◽

2009 ◽

Cited By ~ 94

Author(s):

Stellan Ohlsson

Keyword(s):

Deep Learning

Download Full-text

Intelligence artificielle : le futur de l’Orthodontie ?

Revue d Orthopédie Dento-Faciale ◽

10.1051/odf/2019026 ◽

2019 ◽

Vol 53 (3) ◽

pp. 281-294

Author(s):

Jean-Michel Foucart ◽

Augustin Chavanne ◽

Jérôme Bourriau

Keyword(s):

Big Data ◽

Deep Learning ◽

Intelligence Artificielle ◽

Set Up

Nombreux sont les apports envisagés de l’Intelligence Artificielle (IA) en médecine. En orthodontie, plusieurs solutions automatisées sont disponibles depuis quelques années en imagerie par rayons X (analyse céphalométrique automatisée, analyse automatisée des voies aériennes) ou depuis quelques mois (analyse automatique des modèles numériques, set-up automatisé; CS Model +, Carestream Dental™). L’objectif de cette étude, en deux parties, est d’évaluer la fiabilité de l’analyse automatisée des modèles tant au niveau de leur numérisation que de leur segmentation. La comparaison des résultats d’analyse des modèles obtenus automatiquement et par l’intermédiaire de plusieurs orthodontistes démontre la fiabilité de l’analyse automatique; l’erreur de mesure oscillant, in fine, entre 0,08 et 1,04 mm, ce qui est non significatif et comparable avec les erreurs de mesures inter-observateurs rapportées dans la littérature. Ces résultats ouvrent ainsi de nouvelles perspectives quand à l’apport de l’IA en Orthodontie qui, basée sur le deep learning et le big data, devrait permettre, à moyen terme, d’évoluer vers une orthodontie plus préventive et plus prédictive.

Download Full-text