scholarly journals Style Transfer Using Generative Adversarial Networks for Multi-Site MRI Harmonization

2021 ◽  
Author(s):  
Mengting Liu ◽  
Piyush Maiti ◽  
Sophia Thomopoulos ◽  
Alyssa Zhu ◽  
Yaqiong Chai ◽  
...  
Keyword(s):  
Large Scale ◽  
Transfer Problem ◽  
Large Data ◽  
Generative Adversarial Networks ◽  
Reference Image ◽  
Demographic Information ◽  
Mr Images ◽  
Generative Adversarial Network ◽  
Style Transfer ◽  
Cross Site

AbstractLarge data initiatives and high-powered brain imaging analyses require the pooling of MR images acquired across multiple scanners, often using different protocols. Prospective cross-site harmonization often involves the use of a phantom or traveling subjects. However, as more datasets are becoming publicly available, there is a growing need for retrospective harmonization, pooling data from sites not originally coordinated together. Several retrospective harmonization techniques have shown promise in removing cross-site image variation. However, most unsupervised methods cannot distinguish between image-acquisition based variability and cross-site population variability, so they require that datasets contain subjects or patient groups with similar clinical or demographic information. To overcome this limitation, we consider cross-site MRI image harmonization as a style transfer problem rather than a domain transfer problem. Using a fully unsupervised deep-learning framework based on a generative adversarial network (GAN), we show that MR images can be harmonized by inserting the style information encoded from a reference image directly, without knowing their site/scanner labels a priori. We trained our model using data from five large-scale multi-site datasets with varied demographics. Results demonstrated that our styleencoding model can harmonize MR images, and match intensity profiles, successfully, without relying on traveling subjects. This model also avoids the need to control for clinical, diagnostic, or demographic information. Moreover, we further demonstrated that if we included diverse enough images into the training set, our method successfully harmonized MR images collected from unseen scanners and protocols, suggesting a promising novel tool for ongoing collaborative studies.

scholarly journals Enhanced Magnetic Resonance Image Synthesis with Contrast-Aware Generative Adversarial Networks

2021 ◽  
Vol 7 (8) ◽  
pp. 133
Author(s):  
Jonas Denck ◽  
Jens Guehring ◽  
Andreas Maier ◽  
Eva Rothgang
Keyword(s):  
Magnetic Resonance ◽  
Image Synthesis ◽  
Generative Adversarial Networks ◽  
Data Generation ◽  
Mr Images ◽  
Generative Adversarial Network ◽  
Radiology Training ◽  
Repetition Time ◽  
Echo Time ◽  
Image Orientation

A magnetic resonance imaging (MRI) exam typically consists of the acquisition of multiple MR pulse sequences, which are required for a reliable diagnosis. With the rise of generative deep learning models, approaches for the synthesis of MR images are developed to either synthesize additional MR contrasts, generate synthetic data, or augment existing data for AI training. While current generative approaches allow only the synthesis of specific sets of MR contrasts, we developed a method to generate synthetic MR images with adjustable image contrast. Therefore, we trained a generative adversarial network (GAN) with a separate auxiliary classifier (AC) network to generate synthetic MR knee images conditioned on various acquisition parameters (repetition time, echo time, and image orientation). The AC determined the repetition time with a mean absolute error (MAE) of 239.6 ms, the echo time with an MAE of 1.6 ms, and the image orientation with an accuracy of 100%. Therefore, it can properly condition the generator network during training. Moreover, in a visual Turing test, two experts mislabeled 40.5% of real and synthetic MR images, demonstrating that the image quality of the generated synthetic and real MR images is comparable. This work can support radiologists and technologists during the parameterization of MR sequences by previewing the yielded MR contrast, can serve as a valuable tool for radiology training, and can be used for customized data generation to support AI training.

scholarly journals Learning Representations of Inorganic Materials from Generative Adversarial Networks

Symmetry ◽  
2020 ◽  
Vol 12 (11) ◽  
pp. 1889
Author(s):  
Tiantian Hu ◽  
Hui Song ◽  
Tao Jiang ◽  
Shaobo Li
Keyword(s):  
Large Scale ◽  
Inorganic Materials ◽  
Molecular Composition ◽  
Materials Characterization ◽  
Generative Adversarial Networks ◽  
Feature Engineering ◽  
Material Research ◽  
Generative Adversarial Network ◽  
Characterization Of Materials

The two most important aspects of material research using deep learning (DL) or machine learning (ML) are the characteristics of materials data and learning algorithms, where the proper characterization of materials data is essential for generating accurate models. At present, the characterization of materials based on the molecular composition includes some methods based on feature engineering, such as Magpie and One-hot. Although these characterization methods have achieved significant results in materials research, these methods based on feature engineering cannot guarantee the integrity of materials characterization. One possible approach is to learn the materials characterization via neural networks using the chemical knowledge and implicit composition rules shown in large-scale known materials. This article chooses an adversarial method to learn the composition of atoms using the Generative Adversarial Network (GAN), which makes sense for data symmetry. The total loss value of the discriminator on the test set is reduced from 4.1e13 to 0.3194, indicating that the designed GAN network can well capture the combination of atoms in real materials. We then use the trained discriminator weights for material characterization and predict bandgap, formation energy, critical temperature (Tc) of superconductors on the Open Quantum Materials Database (OQMD), Materials Project (MP), and SuperCond datasets. Experiments show that when using the same predictive model, our proposed method performs better than One-hot and Magpie. This article provides an effective method for characterizing materials based on molecular composition in addition to Magpie, One-hot, etc. In addition, the generator learned in this study generates hypothetical materials with the same distribution as known materials, and these hypotheses can be used as a source for new material discovery.

scholarly journals COVID-GAN+: Estimating Human Mobility Responses to COVID-19 through Spatio-temporal Generative Adversarial Networks with Enhanced Features

2022 ◽  
Vol 13 (2) ◽  
pp. 1-23
Author(s):  
Han Bao ◽  
Xun Zhou ◽  
Yiqun Xie ◽  
Yingxue Zhang ◽  
Yanhua Li
Keyword(s):  
Spatial Heterogeneity ◽  
Real World ◽  
Large Scale ◽  
Census Data ◽  
Human Mobility ◽  
Training Data ◽  
Estimation Accuracy ◽  
Generative Adversarial Networks ◽  
Generative Adversarial Network ◽  
Spatio Temporal

Estimating human mobility responses to the large-scale spreading of the COVID-19 pandemic is crucial, since its significance guides policymakers to give Non-pharmaceutical Interventions, such as closure or reopening of businesses. It is challenging to model due to complex social contexts and limited training data. Recently, we proposed a conditional generative adversarial network (COVID-GAN) to estimate human mobility response under a set of social and policy conditions integrated from multiple data sources. Although COVID-GAN achieves a good average estimation accuracy under real-world conditions, it produces higher errors in certain regions due to the presence of spatial heterogeneity and outliers. To address these issues, in this article, we extend our prior work by introducing a new spatio-temporal deep generative model, namely, COVID-GAN+. COVID-GAN+ deals with the spatial heterogeneity issue by introducing a new spatial feature layer that utilizes the local Moran statistic to model the spatial heterogeneity strength in the data. In addition, we redesign the training objective to learn the estimated mobility changes from historical average levels to mitigate the effects of spatial outliers. We perform comprehensive evaluations using urban mobility data derived from cell phone records and census data. Results show that COVID-GAN+ can better approximate real-world human mobility responses than prior methods, including COVID-GAN.

scholarly journals AUTOMATIC LARGE-SCALE 3D BUILDING SHAPE REFINEMENT USING CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS

2018 ◽  
Vol XLII-2 ◽  
pp. 103-108 ◽  
Author(s):  
K. Bittner ◽  
P. d’Angelo ◽  
M. Körner ◽  
P. Reinartz
Keyword(s):  
Remote Sensing ◽  
Urban Areas ◽  
Large Scale ◽  
Point Clouds ◽  
Research Area ◽  
Sar Interferometry ◽  
Generative Adversarial Networks ◽  
Generative Adversarial Network ◽  
Building Shape ◽  
Object Shapes

<p><strong>Abstract.</strong> Three-dimensional building reconstruction from remote sensing imagery is one of the most difficult and important 3D modeling problems for complex urban environments. The main data sources provided the digital representation of the Earths surface and related natural, cultural, and man-made objects of the urban areas in remote sensing are the <i>digital surface models (DSMs)</i>. The DSMs can be obtained either by <i>light detection and ranging (LIDAR)</i>, SAR interferometry or from stereo images. Our approach relies on automatic global 3D building shape refinement from stereo DSMs using deep learning techniques. This refinement is necessary as the DSMs, which are extracted from image matching point clouds, suffer from occlusions, outliers, and noise. Though most previous works have shown promising results for building modeling, this topic remains an open research area. We present a new methodology which not only generates images with continuous values representing the elevation models but, at the same time, enhances the 3D object shapes, buildings in our case. Mainly, we train a <i>conditional generative adversarial network (cGAN)</i> to generate accurate LIDAR-like DSM height images from the noisy stereo DSM input. The obtained results demonstrate the strong potential of creating large areas remote sensing depth images where the buildings exhibit better-quality shapes and roof forms.</p>

scholarly journals GAN-GL: Generative Adversarial Networks for Glacial Lake Mapping

2021 ◽  
Vol 13 (22) ◽  
pp. 4728
Author(s):  
Hang Zhao ◽  
Meimei Zhang ◽  
Fang Chen
Keyword(s):  
Large Scale ◽  
Glacial Lake ◽  
Background Information ◽  
Generative Adversarial Networks ◽  
High Mountain ◽  
Landsat 8 ◽  
Glacial Lakes ◽  
Ancillary Data ◽  
Generative Adversarial Network ◽  
Adversarial Network

Remote sensing is a powerful tool that provides flexibility and scalability for monitoring and investigating glacial lakes in High Mountain Asia (HMA). However, existing methods for mapping glacial lakes are designed based on a combination of several spectral features and ancillary data (such as the digital elevation model, DEM) to highlight the lake extent and suppress background information. These methods, however, suffer from either the inevitable requirement of post-processing work or the high costs of additional data acquisition. Signifying a key advancement in the deep learning models, a generative adversarial network (GAN) can capture multi-level features and learn the mapping rules in source and target domains using a minimax game between a generator and discriminator. This provides a new and feasible way to conduct large-scale glacial lake mapping. In this work, a complete glacial lake dataset was first created, containing approximately 4600 patches of Landsat-8 OLI images edited in three ways—random cropping, density cropping, and uniform cropping. Then, a GAN model for glacial lake mapping (GAN-GL) was constructed. The GAN-GL consists of two parts—a generator that incorporates a water attention module and an image segmentation module to produce the glacial lake masks, and a discriminator which employs the ResNet-152 backbone to ascertain whether a given pixel belonged to a glacial lake. The model was evaluated using the created glacial lake dataset, delivering a good performance, with an F1 score of 92.17% and IoU of 86.34%. Moreover, compared to the mapping results derived from the global–local iterative segmentation algorithm and random forest for the entire Eastern Himalayas, our proposed model was superior regarding the segmentation of glacial lakes under complex and diverse environmental conditions, in terms of accuracy (precision = 93.19%) and segmentation efficiency. Our model was also very good at detecting small glacial lakes without assistance from ancillary data or human intervention.

scholarly journals Synthesising Facial Macro- and Micro-Expressions Using Reference Guided Style Transfer

2021 ◽  
Vol 7 (8) ◽  
pp. 142
Author(s):  
Chuin Hong Yap ◽  
Ryan Cunningham ◽  
Adrian K. Davison ◽  
Moi Hoon Yap
Keyword(s):  
Evaluation Method ◽  
Performance Metrics ◽  
Synthetic Data ◽  
Future Research ◽  
Reference Image ◽  
Source Image ◽  
Generative Adversarial Network ◽  
Style Transfer ◽  
Facial Movements ◽  
Action Units

Long video datasets of facial macro- and micro-expressions remains in strong demand with the current dominance of data-hungry deep learning methods. There are limited methods of generating long videos which contain micro-expressions. Moreover, there is a lack of performance metrics to quantify the generated data. To address the research gaps, we introduce a new approach to generate synthetic long videos and recommend assessment methods to inspect dataset quality. For synthetic long video generation, we use the state-of-the-art generative adversarial network style transfer method—StarGANv2. Using StarGANv2 pre-trained on the CelebA dataset, we transfer the style of a reference image from SAMM long videos (a facial micro- and macro-expression long video dataset) onto a source image of the FFHQ dataset to generate a synthetic dataset (SAMM-SYNTH). We evaluate SAMM-SYNTH by conducting an analysis based on the facial action units detected by OpenFace. For quantitative measurement, our findings show high correlation on two Action Units (AUs), i.e., AU12 and AU6, of the original and synthetic data with a Pearson’s correlation of 0.74 and 0.72, respectively. This is further supported by evaluation method proposed by OpenFace on those AUs, which also have high scores of 0.85 and 0.59. Additionally, optical flow is used to visually compare the original facial movements and the transferred facial movements. With this article, we publish our dataset to enable future research and to increase the data pool of micro-expressions research, especially in the spotting task.

scholarly journals Study on Optimal Generative Network for Synthesizing Brain Tumor-Segmented MR Images

2020 ◽  
Vol 2020 ◽  
pp. 1-12
Author(s):  
Hyunhee Lee ◽  
Jaechoon Jo ◽  
Heuiseok Lim
Keyword(s):  
Brain Tumor ◽  
Medical Imaging ◽  
Image Synthesis ◽  
Generative Adversarial Networks ◽  
Imaging Data ◽  
Mr Images ◽  
Style Transfer ◽  
Adversarial Networks ◽  
Robust Model ◽  
Privacy Issues

Due to institutional and privacy issues, medical imaging researches are confronted with serious data scarcity. Image synthesis using generative adversarial networks provides a generic solution to the lack of medical imaging data. We synthesize high-quality brain tumor-segmented MR images, which consists of two tasks: synthesis and segmentation. We performed experiments with two different generative networks, the first using the ResNet model, which has significant advantages of style transfer, and the second, the U-Net model, one of the most powerful models for segmentation. We compare the performance of each model and propose a more robust model for synthesizing brain tumor-segmented MR images. Although ResNet produced better-quality images than did U-Net for the same samples, it used a great deal of memory and took much longer to train. U-Net, meanwhile, segmented the brain tumors more accurately than did ResNet.

scholarly journals USING GENERATIVE ADVERSARIAL NETWORKS FOR EXTRACTION OF INSAR SIGNALS FROM LARGE-SCALE SENTINEL-1 INTERFEROGRAMS BY IMPROVING TROPOSPHERIC NOISE CORRECTION

2021 ◽  
Vol V-3-2021 ◽  
pp. 57-64
Author(s):  
B. Ghosh ◽  
M. Haghshenas Haghighi ◽  
M. Motagh ◽  
S. Maghsudi
Keyword(s):  
Large Scale ◽  
Ground Deformation ◽  
Noise Model ◽  
Generative Adversarial Networks ◽  
Tropospheric Delay ◽  
Generative Adversarial Network ◽  
Water Vapour Content ◽  
Adversarial Network ◽  
Adversarial Networks ◽  
Noise Correction

Abstract. Spatiotemporal variations of pressure, temperature, water vapour content in the atmosphere lead to significant delays in interferometric synthetic aperture radar (InSAR) measurements of deformations in the ground. One of the key challenges in increasing the accuracy of ground deformation measurements using InSAR is to produce robust estimates of the tropospheric delay. Tropospheric models like ERA-Interim can be used to estimate the total tropospheric delay in interferograms in remote areas. The problem with using ERA-Interim model for interferogram correction is that after the tropospheric correction, there are still some residuals left in the interferograms, which can be mainly attributed to turbulent troposphere. In this study, we propose a Generative Adversarial Network (GAN) based approach to mitigate the phase delay caused by troposphere. In this method, we implement a noise to noise model, where the network is trained only with the interferograms corrupted by tropospheric noise. We applied the technique over 116 large scale 800 km long interfergrams formed from Sentinel-1 acquisitions covering a period from 25th October, 2014 to 2nd November, 2017 from descending track numbered 108 over Iran. Our approach reduces the root mean square of the phase values of the interferogram 64% compared to those of the original interferogram and by 55% in comparison to the corresponding ERA-Interim corrected version.

scholarly journals CBNWI-50: A Deep Learning Bird Dataset for Image Translation and Resolution Improvement using Generative Adversarial Network

2019 ◽  
Vol 8 (9S) ◽  
pp. 91-102
Keyword(s):  
Deep Learning ◽  
Super Resolution ◽  
Generative Adversarial Networks ◽  
Western India ◽  
Generative Adversarial Network ◽  
Style Transfer ◽  
Adversarial Network ◽  
Image Translation ◽  
Common Birds ◽  
Single Image Super Resolution

Generative Adversarial Networks have gained prominence in a short span of time as they can synthesize images from latent noise by minimizing the adversarial cost function. New variants of GANs have been developed to perform specific tasks using state-of-the-art GAN models, like image translation, single image super resolution, segmentation, classification, style transfer etc. However, a combination of two GANs to perform two different applications in one model has been sparsely explored. Hence, this paper concatenates two GANs and aims to perform Image Translation using Cycle GAN model on bird images and improve their resolution using SRGAN. During the extensive survey, it is observed that most of the deep learning databases on Aves were built using the new world species (i.e. species found in North America). Hence, to bridge this gap, a new Ave database, 'Common Birds of North - Western India' (CBNWI-50), is also proposed in this work.

scholarly journals Scribble-to-Painting Transformation with Multi-Task Generative Adversarial Networks

2019 ◽  
Author(s):  
Jinning Li ◽  
Yexiang Xue
Keyword(s):  
Semantic Segmentation ◽  
Neural Nets ◽  
Experimental Result ◽  
Generative Adversarial Networks ◽  
Neural Net ◽  
Generative Adversarial Network ◽  
Style Transfer ◽  
Adversarial Network ◽  
Classical Image ◽  
Artistic Images

We propose the Dual Scribble-to-Painting Network (DSP-Net), which is able to produce artistic paintings based on user-generated scribbles. In scribble-to-painting transformation, a neural net has to infer additional details of the image, given relatively sparse information contained in the outlines of the scribble. Therefore, it is more challenging than classical image style transfer, in which the information content is reduced from photos to paintings. Inspired by the human cognitive process, we propose a multi-task generative adversarial network, which consists of two jointly trained neural nets -- one for generating artistic images and the other one for semantic segmentation. We demonstrate that joint training on these two tasks brings in additional benefit. Experimental result shows that DSP-Net outperforms state-of-the-art models both visually and quantitatively. In addition, we publish a large dataset for scribble-to-painting transformation.

Sign in / Sign up

Export Citation Format

Share Document