A Dynamic Data Placement Strategy for Hadoop in Heterogeneous Environments

Chia Wei Lee, Kuang Yu Hsieh, Sun Yuan Hsieh, Hung Chang Hsiao

Research output: Contribution to journalArticlepeer-review

43 Citations (Scopus)

Abstract

Cloud computing is a type of parallel distributed computing system that has become a frequently used computer application. MapReduce is an effective programming model used in cloud computing and large-scale data-parallel applications. Hadoop is an open-source implementation of the MapReduce model, and is usually used for data-intensive applications such as data mining and web indexing. The current Hadoop implementation assumes that every node in a cluster has the same computing capacity and that the tasks are data-local, which may increase extra overhead and reduce MapReduce performance. This paper proposes a data placement algorithm to resolve the unbalanced node workload problem. The proposed method can dynamically adapt and balance data stored in each node based on the computing capacity of each node in a heterogeneous Hadoop cluster. The proposed method can reduce data transfer time to achieve improved Hadoop performance. The experimental results show that the dynamic data placement policy can decrease the time of execution and improve Hadoop performance in a heterogeneous cluster.

Original languageEnglish
Pages (from-to)14-22
Number of pages9
JournalBig Data Research
Volume1
DOIs
Publication statusPublished - 2014 Aug 1

All Science Journal Classification (ASJC) codes

  • Management Information Systems
  • Information Systems
  • Computer Science Applications
  • Information Systems and Management

Fingerprint Dive into the research topics of 'A Dynamic Data Placement Strategy for Hadoop in Heterogeneous Environments'. Together they form a unique fingerprint.

Cite this