scholarly journals The LEXIS Platform for Distributed Workflow Execution and Data Management

2022 ◽  
pp. 17-35
Author(s):  
Martin Golasowski ◽  
Jan Martinovič ◽  
Jan Křenek ◽  
Kateřina Slaninová ◽  
Marc Levrier ◽  
...  
Keyword(s):  
Data Management ◽  
Distributed Workflow ◽  
Workflow Execution

Data Management in Scientific Workflows

2012 ◽  
pp. 177-187
Author(s):  
Ewa Deelman ◽  
Ann Chervenak
Keyword(s):  
Data Management ◽  
Gravitational Wave ◽  
Large Scale ◽  
Real Life ◽  
Scientific Workflows ◽  
Workflow Systems ◽  
Automated Generation ◽  
Workflow Execution ◽  
Data Products ◽  
Workflow Planning

Scientific applications such as those in astronomy, earthquake science, gravitational-wave physics, and others have embraced workflow technologies to do large-scale science. Workflows enable researchers to collaboratively design, manage, and obtain results that involve hundreds of thousands of steps, access terabytes of data, and generate similar amounts of intermediate and final data products. Although workflow systems are able to facilitate the automated generation of data products, many issues still remain to be addressed. These issues exist in different forms in the workflow lifecycle. This chapter describes a workflow lifecycle as consisting of a workflow generation phase where the analysis is defined, the workflow planning phase where resources needed for execution are selected, the workflow execution part, where the actual computations take place, and the result, metadata, and provenance storing phase. The authors discuss the issues related to data management at each step of the workflow cycle. They describe challenge problems and illustrate them in the context of real-life applications. They discuss the challenges, possible solutions, and open issues faced when mapping and executing large-scale workflows on current cyberinfrastructure. They particularly emphasize the issues related to the management of data throughout the workflow lifecycle.

scholarly journals A Service-Oriented Architecture enabling dynamic service grouping for optimizing distributed workflow execution

2008 ◽  
Vol 24 (7) ◽  
pp. 720-730 ◽  
Author(s):  
Tristan Glatard ◽  
Johan Montagnat ◽  
David Emsellem ◽  
Diane Lingrand
Keyword(s):  
Service Oriented Architecture ◽  
Distributed Workflow ◽  
Workflow Execution ◽  
Service Oriented

Event-based Distributed Workflow Execution with EVE

1998 ◽  
pp. 427-442 ◽  
Author(s):  
Andreas Geppert ◽  
Dimitrios Tombros
Keyword(s):  
Distributed Workflow ◽  
Workflow Execution ◽  
Event Based

scholarly journals Cost-Effective Resource Provisioning for Real-Time Workflow in Cloud

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-15
Author(s):  
Lei Wu ◽  
Ran Ding ◽  
Zhaohong Jia ◽  
Xuejun Li
Keyword(s):  
Big Data ◽  
Data Management ◽  
Real Time ◽  
Critical Path ◽  
Cost Effective ◽  
Resource Provisioning ◽  
Complex Data ◽  
Volume Data ◽  
Workflow Execution ◽  
Time Requirements

In the era of big data, mining and analysis of the enormous amount of data has been widely used to support decision-making. This complex process including huge-volume data collecting, storage, transmission, and analysis could be modeled as workflow. Meanwhile, cloud environment provides sufficient computing and storage resources for big data management and analytics. Due to the clouds providing the pay-as-you-go pricing scheme, executing a workflow in clouds should pay for the provisioned resources. Thus, cost-effective resource provisioning for workflow in clouds is still a critical challenge. Also, the responses of the complex data management process are usually required to be real-time. Therefore, deadline is the most crucial constraint for workflow execution. In order to address the challenge of cost-effective resource provisioning while meeting the real-time requirements of workflow execution, a resource provisioning strategy based on dynamic programming is proposed to achieve cost-effectiveness of workflow execution in clouds and a critical-path based workflow partition algorithm is presented to guarantee that the workflow can be completed before deadline. Our approach is evaluated by simulation experiments with real-time workflows of different sizes and different structures. The results demonstrate that our algorithm outperforms the existing classical algorithms.

Sign in / Sign up

Export Citation Format

Share Document