scholarly journals Bifrost: highly parallel construction and indexing of colored and compacted de Bruijn graphs

2020 ◽  
Vol 21 (1) ◽  
Author(s):  
Guillaume Holley ◽  
Páll Melsted
Keyword(s):  
Graph Coloring ◽  
Efficient Algorithm ◽  
De Bruijn Graph ◽  
Memory Consumption ◽  
Direct Construction ◽  
De Bruijn Graphs ◽  
Range Of Functions ◽  
De Bruijn ◽  
Parallel Construction ◽  
Memory Efficient

Abstract Memory consumption of de Bruijn graphs is often prohibitive. Most de Bruijn graph-based assemblers reduce the complexity by compacting paths into single vertices, but this is challenging as it requires the uncompacted de Bruijn graph to be available in memory. We present a parallel and memory-efficient algorithm enabling the direct construction of the compacted de Bruijn graph without producing the intermediate uncompacted graph. Bifrost features a broad range of functions, such as indexing, editing, and querying the graph, and includes a graph coloring method that maps each k-mer of the graph to the genomes it occurs in. Availability https://github.com/pmelsted/bifrost

scholarly journals Bifrost – Highly parallel construction and indexing of colored and compacted de Bruijn graphs

2019 ◽  
Author(s):  
Guillaume Holley ◽  
Páll Melsted
Keyword(s):  
High Throughput Sequencing ◽  
Genomic Analysis ◽  
Main Memory ◽  
De Bruijn Graph ◽  
Direct Construction ◽  
De Bruijn Graphs ◽  
Wide Range ◽  
User Data ◽  
De Bruijn ◽  
Colored De Bruijn Graph

AbstractMotivationDe Bruijn graphs are the core data structure for a wide range of assemblers and genome analysis software processing High Throughput Sequencing datasets. For population genomic analysis, the colored de Bruijn graph is often used in order to take advantage of the massive sets of sequenced genomes available for each species. However, memory consumption of tools based on the de Bruijn graph is often prohibitive, due to the high number of vertices, edges or colors in the graph. In order to process large and complex genomes, most short-read assemblers based on the de Bruijn graph paradigm reduce the assembly complexity and memory usage by compacting first all maximal non-branching paths of the graph into single vertices. Yet, de Bruijn graph compaction is challenging as it requires the uncompacted de Bruijn graph to be available in memory.ResultsWe present a new parallel and memory efficient algorithm enabling the direct construction of the compacted de Bruijn graph without producing the intermediate uncompacted de Bruijn graph. Bifrost features a broad range of functions such as sequence querying, storage of user data alongside vertices and graph editing that automatically preserve the compaction property. Bifrost makes full use of the dynamic index efficiency and proposes a graph coloring method efficiently mapping eachk-mer of the graph to the set of genomes in which it occurs. Experimental results show that our algorithm is competitive with state-of-the-art de Bruijn graph compaction and coloring tools. Bifrost was able to build the colored and compacted de Bruijn graph of about 118,000 Salmonella genomes on a mid-class server in about 4 days using 103 GB of main memory.Availabilityhttps://github.com/pmelsted/bifrostavailable with a BSD-2 [email protected]

scholarly journals LOGAN: A framework for LOssless Graph-based ANalysis of high throughput sequence data

2017 ◽  
Author(s):  
Anthony Bolger ◽  
Alisandra Denton ◽  
Marie Bolger ◽  
Björn Usadel
Keyword(s):  
Data Structure ◽  
Data Structures ◽  
Sequence Data ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
Solanum Pennellii ◽  
Desktop Computer ◽  
De Bruijn Graphs ◽  
De Bruijn ◽  
Memory Efficient

AbstractRecent massive growth in the production of sequencing data necessitates matching improve-ments in bioinformatics tools to effectively utilize it. Existing tools suffer from limitations in both scalability and applicability which are inherent to their underlying algorithms and data structures. We identify the key requirements for the ideal data structure for sequence analy-ses: it should be informationally lossless, locally updatable, and memory efficient; requirements which are not met by data structures underlying the major assembly strategies Overlap Layout Consensus and De Bruijn Graphs. We therefore propose a new data structure, the LOGAN graph, which is based on a memory efficient Sparse De Bruijn Graph with routing information. Innovations in storing routing information and careful implementation allow sequence datasets for Escherichia coli (4.6Mbp, 117x coverage), Arabidopsis thaliana (135Mbp, 17.5x coverage) and Solanum pennellii (1.2Gbp, 47x coverage) to be loaded into memory on a desktop computer in seconds, minutes, and hours respectively. Memory consumption is competitive with state of the art alternatives, while losslessly representing the reads in an indexed and updatable form. Both Second and Third Generation Sequencing reads are supported. Thus, the LOGAN graph is positioned to be the backbone for major breakthroughs in sequence analysis such as integrated hybrid assembly, assembly of exceptionally large and repetitive genomes, as well as assembly and representation of pan-genomes.

scholarly journals Building Large Updatable Colored de Bruijn Graphs via Merging

2017 ◽  
Author(s):  
Martin D Muggli ◽  
Bahar Alipanahi ◽  
Christina Boucher
Keyword(s):  
Bloom Filter ◽  
Metagenomic Data ◽  
De Bruijn Graph ◽  
De Bruijn Graphs ◽  
Working Space ◽  
Fold Reduction ◽  
De Bruijn ◽  
Colored De Bruijn Graph ◽  
Public Datasets ◽  
Memory Efficient

MOTIVATION: There exists several massive genomic and metagenomic data collection efforts, including GenomeTrakr and MetaSub, which are routinely updated with new data. To analyze such datasets, memory-efficient methods to construct and store the colored de Bruijn graph have been developed. Yet, a problem that has not been considered is constructing the colored de Bruijn graph in a scalable manner that allows new data to be added without reconstruction. This problem is important for large public datasets as scalability is needed but also the ability to update the construction is also needed. RESULTS: We create a method for constructing and updating the colored de Bruijn graph on a very-large dataset through partitioning the data into smaller subsets, building the colored de Bruijn graph using a FM-index based representation, and succinctly merging these representations to build a single graph. The last step, merging succinctly, is the algorithmic challenge which we solve in this paper. We refer to the resulting method as VariMerge. We validate our approach, and show it produces a three-fold reduction in working space when constructing a colored de Bruijn graph for 8,000 strains. Lastly, we compare VariMerge to other competing methods --- including Vari, Rainbowfish, Mantis, Bloom Filter Trie, the method by Almodaresi(2019) and Multi-BRWT --- and illustrate that VariMerge is the only method that is capable of building the colored de Bruijn graph for 16,000 strains in a manner that allows additional samples to be added. Competing methods either did not scale to this large of a dataset or cannot allow for additions without reconstruction. AVAILABILITY: Our software is available under GPLv3 at https://github.com/cosmo-team/cosmo/tree/VARI-merge.

scholarly journals Scalable Genome Assembly through Parallel de Bruijn Graph Construction for Multiple k-mers

2019 ◽  
Vol 9 (1) ◽  
Author(s):  
Kanak Mahadik ◽  
Christopher Wright ◽  
Milind Kulkarni ◽  
Saurabh Bagchi ◽  
Somali Chaterji
Keyword(s):  
De Novo ◽  
De Bruijn Graph ◽  
High Quality ◽  
De Bruijn Graphs ◽  
Sequencing Technologies ◽  
De Bruijn ◽  
Similar Accuracy ◽  
Valued Graph ◽  
Assembly Algorithms ◽  
Level Parallelism

Abstract Remarkable advancements in high-throughput gene sequencing technologies have led to an exponential growth in the number of sequenced genomes. However, unavailability of highly parallel and scalable de novo assembly algorithms have hindered biologists attempting to swiftly assemble high-quality complex genomes. Popular de Bruijn graph assemblers, such as IDBA-UD, generate high-quality assemblies by iterating over a set of k-values used in the construction of de Bruijn graphs (DBG). However, this process of sequentially iterating from small to large k-values slows down the process of assembly. In this paper, we propose ScalaDBG, which metamorphoses this sequential process, building DBGs for each distinct k-value in parallel. We develop an innovative mechanism to “patch” a higher k-valued graph with contigs generated from a lower k-valued graph. Moreover, ScalaDBG leverages multi-level parallelism, by both scaling up on all cores of a node, and scaling out to multiple nodes simultaneously. We demonstrate that ScalaDBG completes assembling the genome faster than IDBA-UD, but with similar accuracy on a variety of datasets (6.8X faster for one of the most complex genome in our dataset).

scholarly journals Accurate determination of node and arc multiplicities in de bruijn graphs using conditional random fields

2020 ◽  
Vol 21 (1) ◽  
Author(s):  
Aranka Steyaert ◽  
Pieter Audenaert ◽  
Jan Fostier
Keyword(s):  
Genomic Sequence ◽  
Conditional Random Field ◽  
Accurate Determination ◽  
Next Generation Sequencing Data ◽  
De Bruijn Graph ◽  
Sequencing Data ◽  
De Bruijn Graphs ◽  
Sequencing Errors ◽  
Expectation Maximisation ◽  
De Bruijn

Abstract Background De Bruijn graphs are key data structures for the analysis of next-generation sequencing data. They efficiently represent the overlap between reads and hence, also the underlying genome sequence. However, sequencing errors and repeated subsequences render the identification of the true underlying sequence difficult. A key step in this process is the inference of the multiplicities of nodes and arcs in the graph. These multiplicities correspond to the number of times each k-mer (resp. k+1-mer) implied by a node (resp. arc) is present in the genomic sequence. Determining multiplicities thus reveals the repeat structure and presence of sequencing errors. Multiplicities of nodes/arcs in the de Bruijn graph are reflected in their coverage, however, coverage variability and coverage biases render their determination ambiguous. Current methods to determine node/arc multiplicities base their decisions solely on the information in nodes and arcs individually, under-utilising the information present in the sequencing data. Results To improve the accuracy with which node and arc multiplicities in a de Bruijn graph are inferred, we developed a conditional random field (CRF) model to efficiently combine the coverage information within each node/arc individually with the information of surrounding nodes and arcs. Multiplicities are thus collectively assigned in a more consistent manner. Conclusions We demonstrate that the CRF model yields significant improvements in accuracy and a more robust expectation-maximisation parameter estimation. True k-mers can be distinguished from erroneous k-mers with a higher F1 score than existing methods. A C++11 implementation is available at https://github.com/biointec/detoxunder the GNU AGPL v3.0 license.

scholarly journals The Maximum Independent Sets of de Bruijn Graphs of Diameter 3

10.37236/681 ◽  
2011 ◽  
Vol 18 (1) ◽  
Author(s):  
Dustin A. Cartwright ◽  
María Angélica Cueto ◽  
Enrique A. Tobis
Keyword(s):  
Recurrence Relation ◽  
Generating Function ◽  
Independent Sets ◽  
De Bruijn Graph ◽  
Alphabet Size ◽  
De Bruijn Graphs ◽  
Exponential Generating Function ◽  
De Bruijn

The nodes of the de Bruijn graph $B(d,3)$ consist of all strings of length $3$, taken from an alphabet of size $d$, with edges between words which are distinct substrings of a word of length $4$. We give an inductive characterization of the maximum independent sets of the de Bruijn graphs $B(d,3)$ and for the de Bruijn graph of diameter three with loops removed, for arbitrary alphabet size. We derive a recurrence relation and an exponential generating function for their number. This recurrence allows us to construct exponentially many comma-free codes of length 3 with maximal cardinality.

scholarly journals Scalable multiple whole-genome alignment and locally collinear block construction with SibeliaZ

2019 ◽  
Author(s):  
Ilia Minkin ◽  
Paul Medvedev
Keyword(s):  
Single Machine ◽  
De Bruijn Graph ◽  
Genome Alignment ◽  
Whole Genome ◽  
Reconstruction Algorithms ◽  
De Bruijn Graphs ◽  
Significant Step ◽  
De Bruijn ◽  
Whole Genome Alignment ◽  
Computational Resources

AbstractMultiple whole-genome alignment is a challenging problem in bioinformatics. Despite many successes, current methods are not able to keep up with the growing number, length, and complexity of assembled genomes, especially when computational resources are limited. Approaches based on compacted de Bruijn graphs to identify and extend anchors into locally collinear blocks have potential for scalability, but current methods do not scale to mammalian genomes. We present an algorithm, SibeliaZ-LCB, for identifying collinear blocks in closely related genomes based on analysis of the de Bruijn graph. We further incorporate this into a multiple whole-genome alignment pipeline called SibeliaZ. SibeliaZ shows run-time improvements over other methods while maintaining accuracy. On sixteen recently-assembled strains of mice, SibeliaZ runs in under 16 hours on a single machine, while other tools did not run to completion for eight mice within a week. SibeliaZ makes a significant step towards improving scalability of multiple whole-genome alignment and collinear block reconstruction algorithms on a single machine.

Sign in / Sign up

Export Citation Format

Share Document