The load rebalancing problem in distributed file systems

Hsueh Yi Chung, Che Wei Chang, Hung Chang Hsiao, Yu Chang Chao

研究成果: Conference contribution

6 引文 斯高帕斯(Scopus)

摘要

Distributed file systems (DFS) are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions, a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance, that is, the file chunks are not distributed as uniformly as possible in the nodes. Although distributed load balancing algorithms exist in the literature to deal with the load imbalance problem, emerging DFSs in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, we illustrate and define the load rebalancing problem in cloud DFSs. We advocate file systems in clouds shall incorporate decentralized load rebalancing algorithms to eliminate the performance bottleneck and the single point of failure. Simulation results for a potential distributed load balancing algorithm are illustrated. The performance of our proposal implemented in the Hadoop distributed file system is also demonstrated.

原文English
主出版物標題Proceedings - 2012 IEEE International Conference on Cluster Computing, CLUSTER 2012
發行者IEEE Computer Society
頁面117-125
頁數9
ISBN(列印)9780768548074
DOIs
出版狀態Published - 2012 一月 1
事件2012 IEEE International Conference on Cluster Computing, CLUSTER 2012 - Beijing, China
持續時間: 2012 九月 242012 九月 28

出版系列

名字Proceedings - 2012 IEEE International Conference on Cluster Computing, CLUSTER 2012

Other

Other2012 IEEE International Conference on Cluster Computing, CLUSTER 2012
國家China
城市Beijing
期間12-09-2412-09-28

All Science Journal Classification (ASJC) codes

  • Software

指紋 深入研究「The load rebalancing problem in distributed file systems」主題。共同形成了獨特的指紋。

  • 引用此

    Chung, H. Y., Chang, C. W., Hsiao, H. C., & Chao, Y. C. (2012). The load rebalancing problem in distributed file systems. 於 Proceedings - 2012 IEEE International Conference on Cluster Computing, CLUSTER 2012 (頁 117-125). [6337772] (Proceedings - 2012 IEEE International Conference on Cluster Computing, CLUSTER 2012). IEEE Computer Society. https://doi.org/10.1109/CLUSTER.2012.31