Please use this identifier to cite or link to this item: http://buratest.brunel.ac.uk/handle/2438/4982
Title: A heterogeneous computing system for data mining workflows in multi-agent environments
Authors: Luo, P
Lü, K
Huang, R
He, Q
Shi, Z
Keywords: Data mining;Heterogeneous computing;Directed acyclic graph;Multi-agent system environment
Issue Date: 2006
Publisher: Wiley-Blackwell
Citation: Expert Systems 23(5): 258-272, Nov 2006
Abstract: The computing-intensive data mining (DM) process calls for the support of a heterogeneous computing system, which consists of multiple computers with different configurations connected by a high-speed large-area network for increased computational power and resources. The DM process can be described as a multi-phase pipeline process, and in each phase there could be many optional methods. This makes the workflow for DM very complex and it can be modeled only by a directed acyclic graph (DAG). A heterogeneous computing system needs an effective and efficient scheduling framework, which orchestrates all the computing hardware to perform multiple competitive DM workflows. Motivated by the need for a practical solution of the scheduling problem for the DM workflow, this paper proposes a dynamic DAG scheduling algorithm according to the characteristics of an execution time estimation model for DM jobs. Based on an approximate estimation of job execution time, this algorithm first maps DM jobs to machines in a decentralized and diligent (defined in this paper) manner. Then the performance of this initial mapping can be improved through job migrations when necessary. The scheduling heuristic used considers the factors of both the minimal completion time criterion and the critical path in a DAG. We implement this system in an established multi-agent system environment, in which the reuse of existing DM algorithms is achieved by encapsulating them into agents. The system evaluation and its usage in oil well logging analysis are also discussed.
Description: This is an open access article, that can be obtained from the link below- Copyright @ 2006 Wiley-Blackwell
URI: http://bura.brunel.ac.uk/handle/2438/4982
http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0394.2006.00408.x/abstract
DOI: http://dx.doi.org/10.1111/j.1468-0394.2006.00408.x
ISSN: 0266-4720
Appears in Collections:Business and Management
Brunel Business School Research Papers

Files in This Item:
There are no files associated with this item.


Items in BURA are protected by copyright, with all rights reserved, unless otherwise indicated.