FedLoop: Looping on federated MapReduce

Chun Yu Wang, Tzu Li Tai, Kuan Chieh Huang, Tse En Liu, Jyh Biau Chang, Ce Kuen Shieh

研究成果: Conference contribution

2 引文 斯高帕斯(Scopus)

摘要

The challenges of the Big Data era has motivated many organizations to turn towards distributed, large-scale processing platforms to deal with their data. Map Reduce, and its open-source implementation, Hadoop, has grown to be highly popular with its successful programming model for simplified cluster processing. As a result, many organizations deploy their own Map Reduce/Hadoop clusters to store and process large amounts of useful data. This multicluster setting is gradually growing attention. Numerous previous works have researched on how to execute Map Reduce across geographically distributed data in this setting. However, an important class of applications have not been explored for multicluster Map Reduce: iterative computation. In this paper, we propose Fed Loop, a composite system aimed at providing iterative Map Reduce computation for geographically distributed data in multicluster settings. Fed Loop is capable of transparently executing both iterative and non-iterative Map Reduce jobs on either a single cluster or multiple clusters. For our performance evaluation, two well-known iterative algorithms was executed over 4 independent clusters (16 physical nodes in total) using Fed Loop: K-Means and Page Rank. Results helped us discover how different iterative applications may differ in execution efficiency for mutlicluster environments and how iterative multicluster computation systems like Fed Loop can be optimized.

原文English
主出版物標題Proceedings - 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014
發行者Institute of Electrical and Electronics Engineers Inc.
頁面755-762
頁數8
ISBN(電子)9781479965137
DOIs
出版狀態Published - 2015 一月 15
事件13th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014 - Beijing, China
持續時間: 2014 九月 242014 九月 26

出版系列

名字Proceedings - 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014

Other

Other13th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014
國家China
城市Beijing
期間14-09-2414-09-26

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Safety, Risk, Reliability and Quality
  • Computer Networks and Communications

指紋 深入研究「FedLoop: Looping on federated MapReduce」主題。共同形成了獨特的指紋。

  • 引用此

    Wang, C. Y., Tai, T. L., Huang, K. C., Liu, T. E., Chang, J. B., & Shieh, C. K. (2015). FedLoop: Looping on federated MapReduce. 於 Proceedings - 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014 (頁 755-762). [7011323] (Proceedings - 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/TrustCom.2014.99