scholarly journals X-MOL: large-scale pre-training for molecular understanding and diverse molecular analysis

2020 ◽  
Author(s):  
Dongyu Xue ◽  
Han Zhang ◽  
Dongling Xiao ◽  
Yukang Gong ◽  
Guohui Chuai ◽  
...  
Keyword(s):  
Molecular Analysis ◽  
In Silico ◽  
Large Scale ◽  
De Novo ◽  
Representation Learning ◽  
Training Data ◽  
Fine Tuning ◽  
Model Interpretation ◽  
Unlabelled Data ◽  
Super Computing

AbstractIn silico modelling and analysis of small molecules substantially accelerates the process of drug development. Representing and understanding molecules is the fundamental step for various in silico molecular analysis tasks. Traditionally, these molecular analysis tasks have been investigated individually and separately. In this study, we presented X-MOL, which applies large-scale pre-training technology on 1.1 billion molecules for molecular understanding and representation, and then, carefully designed fine-tuning was performed to accommodate diverse downstream molecular analysis tasks, including molecular property prediction, chemical reaction analysis, drug-drug interaction prediction, de novo generation of molecules and molecule optimization. As a result, X-MOL was proven to achieve state-of-the-art results on all these molecular analysis tasks with good model interpretation ability. Collectively, taking advantage of super large-scale pre-training data and super-computing power, our study practically demonstrated the utility of the idea of “mass makes miracles” in molecular representation learning and downstream in silico molecular analysis, indicating the great potential of using large-scale unlabelled data with carefully designed pre-training and fine-tuning strategies to unify existing molecular analysis tasks and substantially enhance the performance of each task.

scholarly journals Accurate long-read de novo assembly evaluation with Inspector

2021 ◽  
Vol 22 (1) ◽  
Author(s):  
Yu Chen ◽  
Yixin Zhang ◽  
Amy Y. Wang ◽  
Min Gao ◽  
Zechen Chong
Keyword(s):  
Genome Assembly ◽  
De Novo Assembly ◽  
In Silico ◽  
Large Scale ◽  
De Novo ◽  
Small Scale ◽  
De Novo Genome Assembly ◽  
Consensus Sequences ◽  
Assembly Evaluation ◽  
Long Read

AbstractLong-read de novo genome assembly continues to advance rapidly. However, there is a lack of effective tools to accurately evaluate the assembly results, especially for structural errors. We present Inspector, a reference-free long-read de novo assembly evaluator which faithfully reports types of errors and their precise locations. Notably, Inspector can correct the assembly errors based on consensus sequences derived from raw reads covering erroneous regions. Based on in silico and long-read assembly results from multiple long-read data and assemblers, we demonstrate that in addition to providing generic metrics, Inspector can accurately identify both large-scale and small-scale assembly errors.

Combining Self-supervised Learning and Active Learning for Disfluency Detection

2022 ◽  
Vol 21 (3) ◽  
pp. 1-25
Author(s):  
Shaolei Wang ◽  
Zhongyuan Wang ◽  
Wanxiang Che ◽  
Sendong Zhao ◽  
Ting Liu
Keyword(s):  
Neural Network ◽  
Active Learning ◽  
Supervised Learning ◽  
Large Scale ◽  
Training Data ◽  
Fine Tuning ◽  
Training Dataset ◽  
Performance Gap ◽  
Annotation Costs ◽  
Trained Neural Network

Spoken language is fundamentally different from the written language in that it contains frequent disfluencies or parts of an utterance that are corrected by the speaker. Disfluency detection (removing these disfluencies) is desirable to clean the input for use in downstream NLP tasks. Most existing approaches to disfluency detection heavily rely on human-annotated data, which is scarce and expensive to obtain in practice. To tackle the training data bottleneck, in this work, we investigate methods for combining self-supervised learning and active learning for disfluency detection. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled data and propose two self-supervised pre-training tasks: (i) a tagging task to detect the added noisy words and (ii) sentence classification to distinguish original sentences from grammatically incorrect sentences. We then combine these two tasks to jointly pre-train a neural network. The pre-trained neural network is then fine-tuned using human-annotated disfluency detection training data. The self-supervised learning method can capture task-special knowledge for disfluency detection and achieve better performance when fine-tuning on a small annotated dataset compared to other supervised methods. However, limited in that the pseudo training data are generated based on simple heuristics and cannot fully cover all the disfluency patterns, there is still a performance gap compared to the supervised models trained on the full training dataset. We further explore how to bridge the performance gap by integrating active learning during the fine-tuning process. Active learning strives to reduce annotation costs by choosing the most critical examples to label and can address the weakness of self-supervised learning with a small annotated dataset. We show that by combining self-supervised learning with active learning, our model is able to match state-of-the-art performance with just about 10% of the original training data on both the commonly used English Switchboard test set and a set of in-house annotated Chinese data.

scholarly journals A general approach for improving deep learning-based medical relation extraction using a pre-trained model and fine-tuning

Database ◽  
2019 ◽  
Vol 2019 ◽  
Author(s):  
Tao Chen ◽  
Mingfen Wu ◽  
Hexi Li
Keyword(s):  
Deep Learning ◽  
Large Scale ◽  
Relation Extraction ◽  
Training Model ◽  
Biomedical Literature ◽  
Training Data ◽  
Fine Tuning ◽  
Learning Approaches ◽  
Additional Time ◽  
Clinical Records

Abstract The automatic extraction of meaningful relations from biomedical literature or clinical records is crucial in various biomedical applications. Most of the current deep learning approaches for medical relation extraction require large-scale training data to prevent overfitting of the training model. We propose using a pre-trained model and a fine-tuning technique to improve these approaches without additional time-consuming human labeling. Firstly, we show the architecture of Bidirectional Encoder Representations from Transformers (BERT), an approach for pre-training a model on large-scale unstructured text. We then combine BERT with a one-dimensional convolutional neural network (1d-CNN) to fine-tune the pre-trained model for relation extraction. Extensive experiments on three datasets, namely the BioCreative V chemical disease relation corpus, traditional Chinese medicine literature corpus and i2b2 2012 temporal relation challenge corpus, show that the proposed approach achieves state-of-the-art results (giving a relative improvement of 22.2, 7.77, and 38.5% in F1 score, respectively, compared with a traditional 1d-CNN classifier). The source code is available at https://github.com/chentao1999/MedicalRelationExtraction.

scholarly journals Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

2021 ◽  
Vol 4 (1) ◽  
Author(s):  
Laila Rasmy ◽  
Yang Xiang ◽  
Ziqian Xie ◽  
Cui Tao ◽  
Degui Zhi
Keyword(s):  
Deep Learning ◽  
Electronic Health Records ◽  
Large Scale ◽  
Training Data ◽  
Fine Tuning ◽  
Disease Prediction ◽  
Operating Characteristics ◽  
Health Records ◽  
Clinical Databases ◽  
Electronic Health

AbstractDeep learning (DL)-based predictive models from electronic health records (EHRs) deliver impressive performance in many clinical tasks. Large training cohorts, however, are often required by these models to achieve high accuracy, hindering the adoption of DL-based models in scenarios with limited training data. Recently, bidirectional encoder representations from transformers (BERT) and related models have achieved tremendous successes in the natural language processing domain. The pretraining of BERT on a very large training corpus generates contextualized embeddings that can boost the performance of models trained on smaller datasets. Inspired by BERT, we propose Med-BERT, which adapts the BERT framework originally developed for the text domain to the structured EHR domain. Med-BERT is a contextualized embedding model pretrained on a structured EHR dataset of 28,490,650 patients. Fine-tuning experiments showed that Med-BERT substantially improves the prediction accuracy, boosting the area under the receiver operating characteristics curve (AUC) by 1.21–6.14% in two disease prediction tasks from two clinical databases. In particular, pretrained Med-BERT obtains promising performances on tasks with small fine-tuning training sets and can boost the AUC by more than 20% or obtain an AUC as high as a model trained on a training set ten times larger, compared with deep learning models without Med-BERT. We believe that Med-BERT will benefit disease prediction studies with small local training datasets, reduce data collection expenses, and accelerate the pace of artificial intelligence aided healthcare.

scholarly journals Comprehensive Fitness Landscape of a Multi-Geometry Protein Capsid Informs Machine Learning Models of Assembly

2021 ◽  
Author(s):  
Daniel D. Brauer ◽  
Celine B. Santiago ◽  
Zoe N. Merz ◽  
Esther McCarthy ◽  
Danielle Tullman-Ercek ◽  
...  
Keyword(s):  
Machine Learning ◽  
In Silico ◽  
Quaternary Structure ◽  
De Novo ◽  
Fitness Landscape ◽  
Machine Learning Algorithms ◽  
Training Data ◽  
Particle Assembly ◽  
Complex Particle ◽  
Self Assembled

Virus-like particles (VLPs) are non-infections viral-derived nanomaterials poised for biotechnological applications due to their well-defined, modular self-assembling architecture. Although progress has been made in understanding the complex effects that mutations may have on VLPs, nuanced understanding of the influence particle mutability has on quaternary structure has yet to be achieved. Here, we generate and compare the apparent fitness landscapes of two capsid geometries (T=3 and T=1 icosahedral) of the bacteriophage MS2 VLP. We find significant shifts in mutability at the symmetry interfaces of the T=1 capsid when compared to the wildtype T=3 assembly. Furthermore, we use the generated landscapes to benchmark the performance of in silico mutational scanning tools in capturing the effect of missense mutation on complex particle assembly. Finding that predicted stability effects correlated relatively poorly with assembly phenotype, we used a combination of de novo features in tandem with in silico results to train machine learning algorithms for the classification of variant effects on assembly. Our findings not only reveal ways that assembly geometry affects the mutable landscape of a self-assembled particle, but also establish a template for the generation of predictive mutational models of self-assembled capsids using minimal empirical training data.

scholarly journals Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

2021 ◽  
Vol 9 ◽  
pp. 978-994
Author(s):  
Emanuele Bugliarello ◽  
Ryan Cotterell ◽  
Naoaki Okazaki ◽  
Desmond Elliott
Keyword(s):  
Language Processing ◽  
Large Scale ◽  
Meta Analysis ◽  
Training Data ◽  
Fine Tuning ◽  
Controlled Experiments ◽  
Unified Framework ◽  
Massive Models ◽  
Vision And Language ◽  
And Task

Abstract Large-scale pretraining and task-specific fine- tuning is now the standard methodology for many tasks in computer vision and natural language processing. Recently, a multitude of methods have been proposed for pretraining vision and language BERTs to tackle challenges at the intersection of these two key areas of AI. These models can be categorized into either single-stream or dual-stream encoders. We study the differences between these two categories, and show how they can be unified under a single theoretical framework. We then conduct controlled experiments to discern the empirical differences between five vision and language BERTs. Our experiments show that training data and hyperparameters are responsible for most of the differences between the reported results, but they also reveal that the embedding layer plays a crucial role in these massive models.

scholarly journals Transfer Learning for Inference of Metastatic Origin from Whole Slide Histology

2021 ◽  
Author(s):  
Geoffrey F. Schau ◽  
Hassan Ghani ◽  
Erik A. Burlingame ◽  
Guillaume Thibault ◽  
Joe W. Gray ◽  
...  
Keyword(s):  
Deep Learning ◽  
Transfer Learning ◽  
Primary Tumor ◽  
Large Scale ◽  
Metastatic Cancer ◽  
Clinical Diagnostics ◽  
Training Data ◽  
Fine Tuning ◽  
Learning Approach ◽  
Whole Slide Images

AbstractAccurate diagnosis of metastatic cancer is essential for prescribing optimal control strategies to halt further spread of metastasizing disease. While pathological inspection aided by immunohistochemistry staining provides a valuable gold standard for clinical diagnostics, deep learning methods have emerged as powerful tools for identifying clinically relevant features of whole slide histology relevant to a tumor’s metastatic origin. Although deep learning models require significant training data to learn effectively, transfer learning paradigms provide mechanisms to circumvent limited training data by first training a model on related data prior to fine-tuning on smaller data sets of interest. In this work we propose a transfer learning approach that trains a convolutional neural network to infer the metastatic origin of tumor tissue from whole slide images of hematoxylin and eosin (H&E) stained tissue sections and illustrate the advantages of pre-training network on whole slide images of primary tumor morphology. We further characterize statistical dissimilarity between primary and metastatic tumors of various indications on patch-level images to highlight limitations of our indication-specific transfer learning approach. Using a primary-to-metastatic transfer learning approach, we achieved mean class-specific areas under receiver operator characteristics curve (AUROC) of 0.779, which outperformed comparable models trained on only images of primary tumor (mean AUROC of 0.691) or trained on only images of metastatic tumor (mean AUROC of 0.675), supporting the use of large scale primary tumor imaging data in developing computer vision models to characterize metastatic origin of tumor lesions.

scholarly journals Cross-modal Common Representation Learning by Hybrid Transfer Network

2017 ◽  
Author(s):  
Xin Huang ◽  
Yuxin Peng ◽  
Mingkuan Yuan
Keyword(s):  
Domain Knowledge ◽  
Large Scale ◽  
Representation Learning ◽  
Training Data ◽  
Target Domain ◽  
Retrieval Task ◽  
Source Domain ◽  
Useful Knowledge ◽  
Semantic Correlation ◽  
Common Representation

DNN-based cross-modal retrieval is a research hotspot to retrieve across different modalities as image and text, but existing methods often face the challenge of insufficient cross-modal training data. In single-modal scenario, similar problem is usually relieved by transferring knowledge from large-scale auxiliary datasets (as ImageNet). Knowledge from such single-modal datasets is also very useful for cross-modal retrieval, which can provide rich general semantic information that can be shared across different modalities. However, it is challenging to transfer useful knowledge from single-modal (as image) source domain to cross-modal (as image/text) target domain. Knowledge in source domain cannot be directly transferred to both two different modalities in target domain, and the inherent cross-modal correlation contained in target domain provides key hints for cross-modal retrieval which should be preserved during transfer process. This paper proposes Cross-modal Hybrid Transfer Network (CHTN) with two subnetworks: Modal-sharing transfer subnetwork utilizes the modality in both source and target domains as a bridge, for transferring knowledge to both two modalities simultaneously; Layer-sharing correlation subnetwork preserves the inherent cross-modal semantic correlation to further adapt to cross-modal retrieval task. Cross-modal data can be converted to common representation by CHTN for retrieval, and comprehensive experiment on 3 datasets shows its effectiveness.

scholarly journals Deep Embedding Sentiment Analysis on Product Reviews Using Naive Bayesian Classifier

2019 ◽  
pp. 858-864
Author(s):  
Nukabathini Mary Saroj Sahithya ◽  
Manda Prathyusha ◽  
Nakkala Rachana ◽  
Perikala Priyanka ◽  
P. J. Jyothi
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Large Scale ◽  
Opinion Mining ◽  
Machine Learning Algorithms ◽  
Sentiment Classification ◽  
Training Data ◽  
Fine Tuning ◽  
Product Reviews ◽  
Deep Embedding

Product reviews are valuable for upcoming buyers in helping them make decisions. To this end, different opinion mining techniques have been proposed, where judging a review sentence�s orientation (e.g. positive or negative) is one of their key challenges. Recently, deep learning has emerged as an effective means for solving sentiment classification problems. Deep learning is a class of machine learning algorithms that learn in supervised and unsupervised manners. A neural network intrinsically learns a useful representation automatically without human efforts. However, the success of deep learning highly relies on the large-scale training data. We propose a novel deep learning framework for product review sentiment classification which employs prevalently available ratings supervision signals. The framework consists of two steps: (1) learning a high-level representation (an embedding space) which captures the general sentiment distribution of sentences through rating information; (2) adding a category layer on top of the embedding layer and use labelled sentences for supervised fine-tuning. We explore two kinds of low-level network structure for modelling review sentences, namely, convolutional function extractors and long temporary memory. Convolutional layer is the core building block of a CNN and it consists of kernels. Applications are image and video recognition, natural language processing, image classification

Sign in / Sign up

Export Citation Format

Share Document