scholarly journals Combating the hate speech in Thai textual memes

2021 ◽  
Vol 21 (3) ◽  
pp. 1493
Author(s):  
Lawankorn Mookdarsanit ◽  
Pakpoom Mookdarsanit
Keyword(s):  
Neural Network ◽  
Language Processing ◽  
Hate Speech ◽  
Language Model ◽  
Research Problem ◽  
Language Understanding ◽  
Text Localization ◽  
Scene Text ◽  
New Research ◽  
Optical Recognition

<span>Thai textual memes have been popular in social media, as a form of image information summarization. Unfortunately, many memes contain some hateful content that easily causes the controversy in Thailand. </span><span>For global protection, t</span><span>he </span><em><span>Hateful Memes Challenge</span></em><span> is also provided by </span><em><span>Facebook AI</span></em><span> to enable researchers to compete their algorithms for combating the hate speech on memes as one of </span><em><span>NeurIPS’20</span></em><span> competitions. As well as in Thailand, this paper introduces the Thai textual meme detection as a new research problem in Thai natural language processing (Thai-NLP) that is the settlement of transmission linkage between scene text localization, Thai optical recognition (Thai-OCR) and language understanding. From the results, both regular and irregular text position can be localized by one-stage detection pipeline. More scene text can be augmented by different resolution and rotation. The accuracy of Thai-OCR using convolutional neural network (CNN) can be improved by recurrent neural network (RNN). Since misspelling Thai words are frequently used in social, this paper categorizes them as synonyms to train on multi-task pre-trained language model. </span>

A Two-Level Recurrent Neural Network Language Model Based on the Continuous Bag-of-Words Model for Sentence Classification

2019 ◽  
Vol 28 (01) ◽  
pp. 1950002
Author(s):  
Yo Han Lee ◽  
Dong W. Kim ◽  
Myo Taeg Lim
Keyword(s):  
Neural Network ◽  
Language Processing ◽  
Recurrent Neural Network ◽  
Language Model ◽  
Basic Structure ◽  
Bag Of Words ◽  
Fixed Length ◽  
Sentence Classification ◽  
Network Language ◽  
Vector Representations

In this paper, a new two-level recurrent neural network language model (RNNLM) based on the continuous bag-of-words (CBOW) model for application to sentence classification is presented. The vector representations of words learned by a neural network language model have been shown to carry semantic sentiment and are useful in various natural language processing tasks. A disadvantage of CBOW is that it only considers the fixed length of a context because its basic structure is a neural network with a fixed length of input. In contrast, the RNNLM does not have a size limit for a context but only considers the previous context’s words. Therefore, the advantage of RNNLM is complementary to the disadvantage of CBOW. Herein, our proposed model encodes many linguistic patterns and improves upon sentiment analysis and question classification benchmarks compared to previously reported methods.

scholarly journals Where do Clinical Language Models Break Down? A Critical Behavioural Exploration of the ClinicalBERT Deep Transformer Model

2021 ◽  
Vol 6 (1) ◽  
pp. 1-4
Author(s):  
Alexander MacLean ◽  
Alexander Wong
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Language Model ◽  
Language Models ◽  
Clinical Knowledge ◽  
Language Understanding ◽  
Improved Performance ◽  
Transformer Model ◽  
Clinical Domain

The introduction of Bidirectional Encoder Representations from Transformers (BERT) was a major breakthrough for transfer learning in natural language processing, enabling state-of-the-art performance across a large variety of complex language understanding tasks. In the realm of clinical language modeling, the advent of BERT led to the creation of ClinicalBERT, a state-of-the-art deep transformer model pretrained on a wealth of patient clinical notes to facilitate for downstream predictive tasks in the clinical domain. While ClinicalBERT has been widely leveraged by the research community as the foundation for building clinical domain-specific predictive models given its overall improved performance in the Medical Natural Language inference (MedNLI) challenge compared to the seminal BERT model, the fine-grained behaviour and intricacies of this popular clinical language model has not been well-studied. Without this deeper understanding, it is very challenging to understand where ClinicalBERT does well given its additional exposure to clinical knowledge, where it doesn't, and where it can be improved in a meaningful manner. Motivated to garner a deeper understanding, this study presents a critical behaviour exploration of the ClinicalBERT deep transformer model using MedNLI challenge dataset to better understanding the following intricacies: 1) decision-making similarities between ClinicalBERT and BERT (leverage a new metric we introduce called Model Alignment), 2) where ClinicalBERT holds advantages over BERT given its clinical knowledge exposure, and 3) where ClinicalBERT struggles when compared to BERT. The insights gained about the behaviour of ClinicalBERT will help guide towards new directions for designing and training clinical language models in a way that not only addresses the remaining gaps and facilitates for further improvements in clinical language understanding performance, but also highlights the limitation and boundaries of use for such models.

scholarly journals Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking

2020 ◽  
Vol 34 (05) ◽  
pp. 8766-8774 ◽  
Author(s):  
Timo Schick ◽  
Hinrich Schütze
Keyword(s):  
Neural Network ◽  
Natural Language Processing ◽  
Language Processing ◽  
Deep Neural Network ◽  
Language Model ◽  
Language Modeling ◽  
Fine Tuning ◽  
Language Models ◽  
Network Architectures ◽  
Semantic Properties

Pretraining deep neural network architectures with a language modeling objective has brought large improvements for many natural language processing tasks. Exemplified by BERT, a recently proposed such architecture, we demonstrate that despite being trained on huge amounts of data, deep language models still struggle to understand rare words. To fix this problem, we adapt Attentive Mimicking, a method that was designed to explicitly learn embeddings for rare words, to deep language models. In order to make this possible, we introduce one-token approximation, a procedure that enables us to use Attentive Mimicking even when the underlying language model uses subword-based tokenization, i.e., it does not assign embeddings to all words. To evaluate our method, we create a novel dataset that tests the ability of language models to capture semantic properties of words without any task-specific fine-tuning. Using this dataset, we show that adding our adapted version of Attentive Mimicking to BERT does substantially improve its understanding of rare words.

scholarly journals Multimodal Hate Speech Detection in Greek Social Media

2021 ◽  
Vol 5 (7) ◽  
pp. 34
Author(s):  
Konstantinos Perifanos ◽  
Dionysis Goutsos
Keyword(s):  
Social Media ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Hate Speech ◽  
Language Model ◽  
Fine Tuning ◽  
Accuracy Score ◽  
Speech Detection ◽  
Online Social Media

Hateful and abusive speech presents a major challenge for all online social media platforms. Recent advances in Natural Language Processing and Natural Language Understanding allow for more accurate detection of hate speech in textual streams. This study presents a new multimodal approach to hate speech detection by combining Computer Vision and Natural Language processing models for abusive context detection. Our study focuses on Twitter messages and, more specifically, on hateful, xenophobic, and racist speech in Greek aimed at refugees and migrants. In our approach, we combine transfer learning and fine-tuning of Bidirectional Encoder Representations from Transformers (BERT) and Residual Neural Networks (Resnet). Our contribution includes the development of a new dataset for hate speech classification, consisting of tweet IDs, along with the code to obtain their visual appearance, as they would have been rendered in a web browser. We have also released a pre-trained Language Model trained on Greek tweets, which has been used in our experiments. We report a consistently high level of accuracy (accuracy score = 0.970, f1-score = 0.947 in our best model) in racist and xenophobic speech detection.

scholarly journals Extracting Parallel Sentences from Nonparallel Corpora Using Parallel Hierarchical Attention Network

2020 ◽  
Vol 2020 ◽  
pp. 1-9
Author(s):  
Shaolin Zhu ◽  
Yong Yang ◽  
Chun Xu
Keyword(s):  
Neural Network ◽  
Natural Language Processing ◽  
Natural Language ◽  
Machine Translation ◽  
Language Processing ◽  
State Of The Art ◽  
Research Problem ◽  
Shared Task ◽  
Translation Systems

Collecting parallel sentences from nonparallel data is a long-standing natural language processing research problem. In particular, parallel training sentences are very important for the quality of machine translation systems. While many existing methods have shown encouraging results, they cannot learn various alignment weights in parallel sentences. To address this issue, we propose a novel parallel hierarchical attention neural network which encodes monolingual sentences versus bilingual sentences and construct a classifier to extract parallel sentences. In particular, our attention mechanism structure can learn different alignment weights of words in parallel sentences. Experimental results show that our model can obtain state-of-the-art performance on the English-French, English-German, and English-Chinese dataset of BUCC 2017 shared task about parallel sentences’ extraction.

Fast Neural Network Engine for Natural Science Language Processing: A Drug-Search Case.

2020 ◽  
Author(s):  
Vadim V. Korolev ◽  
Artem Mitrofanov ◽  
Kirill Karpov ◽  
Valery Tkachenko
Keyword(s):  
Neural Network ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Natural Science ◽  
Therapeutic Agent ◽  
Semantic Relations ◽  
Chemical Data ◽  
Processing Methods ◽  
Modern Natural

The main advantage of modern natural language processing methods is a possibility to turn an amorphous human-readable task into a strict mathematic form. That allows to extract chemical data and insights from articles and to find new semantic relations. We propose a universal engine for processing chemical and biological texts. We successfully tested it on various use-cases and applied to a case of searching a therapeutic agent for a COVID-19 disease by analyzing PubMed archive.

scholarly journals Identifying protein subcellular localisation in scientific literature using bidirectional deep recurrent neural network

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Rakesh David ◽  
Rhys-Joshua D. Menezes ◽  
Jan De Klerk ◽  
Ian R. Castleden ◽  
Cornelia M. Hooper ◽  
...  
Keyword(s):  
Neural Network ◽  
Language Processing ◽  
Data Dissemination ◽  
Short Term Memory ◽  
Biological Data ◽  
Experimental Methodology ◽  
Subcellular Localisation ◽  
Crop Species ◽  
Deep Recurrent Neural Network ◽  
Functional Features

AbstractThe increased diversity and scale of published biological data has to led to a growing appreciation for the applications of machine learning and statistical methodologies to gain new insights. Key to achieving this aim is solving the Relationship Extraction problem which specifies the semantic interaction between two or more biological entities in a published study. Here, we employed two deep neural network natural language processing (NLP) methods, namely: the continuous bag of words (CBOW), and the bi-directional long short-term memory (bi-LSTM). These methods were employed to predict relations between entities that describe protein subcellular localisation in plants. We applied our system to 1700 published Arabidopsis protein subcellular studies from the SUBA manually curated dataset. The system combines pre-processing of full-text articles in a machine-readable format with relevant sentence extraction for downstream NLP analysis. Using the SUBA corpus, the neural network classifier predicted interactions between protein name, subcellular localisation and experimental methodology with an average precision, recall rate, accuracy and F1 scores of 95.1%, 82.8%, 89.3% and 88.4% respectively (n = 30). Comparable scoring metrics were obtained using the CropPAL database as an independent testing dataset that stores protein subcellular localisation in crop species, demonstrating wide applicability of prediction model. We provide a framework for extracting protein functional features from unstructured text in the literature with high accuracy, improving data dissemination and unlocking the potential of big data text analytics for generating new hypotheses.

Learning emotional word embeddings for sentiment analysis

2021 ◽  
pp. 1-13
Author(s):  
Qingtian Zeng ◽  
Xishi Zhao ◽  
Xiaohui Hu ◽  
Hua Duan ◽  
Zhongying Zhao ◽  
...  
Keyword(s):  
Sentiment Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Research Problem ◽  
Emotional Word ◽  
Classification Model ◽  
Data Sets ◽  
Word Embeddings ◽  
Real World Data ◽  
Text Documents

Word embeddings have been successfully applied in many natural language processing tasks due to its their effectiveness. However, the state-of-the-art algorithms for learning word representations from large amounts of text documents ignore emotional information, which is a significant research problem that must be addressed. To solve the above problem, we propose an emotional word embedding (EWE) model for sentiment analysis in this paper. This method first applies pre-trained word vectors to represent document features using two different linear weighting methods. Then, the resulting document vectors are input to a classification model and used to train a text sentiment classifier, which is based on a neural network. In this way, the emotional polarity of the text is propagated into the word vectors. The experimental results on three kinds of real-world data sets demonstrate that the proposed EWE model achieves superior performances on text sentiment prediction, text similarity calculation, and word emotional expression tasks compared to other state-of-the-art models.

scholarly journals Multi-Transformer: A New Neural Network-Based Architecture for Forecasting S&P Volatility

Mathematics ◽  
2021 ◽  
Vol 9 (15) ◽  
pp. 1794
Author(s):  
Eduardo Ramos-Pérez ◽  
Pablo J. Alonso-González ◽  
José Javier Núñez-Velázquez
Keyword(s):  
Neural Network ◽  
Language Processing ◽  
Short Term Memory ◽  
Risk Measures ◽  
Hybrid Models ◽  
Stock Volatility ◽  
Management Actions ◽  
Equity Risk ◽  
Hedging Strategies ◽  
Volatility Models

Events such as the Financial Crisis of 2007–2008 or the COVID-19 pandemic caused significant losses to banks and insurance entities. They also demonstrated the importance of using accurate equity risk models and having a risk management function able to implement effective hedging strategies. Stock volatility forecasts play a key role in the estimation of equity risk and, thus, in the management actions carried out by financial institutions. Therefore, this paper has the aim of proposing more accurate stock volatility models based on novel machine and deep learning techniques. This paper introduces a neural network-based architecture, called Multi-Transformer. Multi-Transformer is a variant of Transformer models, which have already been successfully applied in the field of natural language processing. Indeed, this paper also adapts traditional Transformer layers in order to be used in volatility forecasting models. The empirical results obtained in this paper suggest that the hybrid models based on Multi-Transformer and Transformer layers are more accurate and, hence, they lead to more appropriate risk measures than other autoregressive algorithms or hybrid models based on feed forward layers or long short term memory cells.

Sign in / Sign up

Export Citation Format

Share Document