FedLoop: Looping on federated MapReduce

Chun Yu Wang, Tzu Li Tai, Kuan Chieh Huang, Tse En Liu, Jyh Biau Chang, Ce Kuen Shieh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

The challenges of the Big Data era has motivated many organizations to turn towards distributed, large-scale processing platforms to deal with their data. Map Reduce, and its open-source implementation, Hadoop, has grown to be highly popular with its successful programming model for simplified cluster processing. As a result, many organizations deploy their own Map Reduce/Hadoop clusters to store and process large amounts of useful data. This multicluster setting is gradually growing attention. Numerous previous works have researched on how to execute Map Reduce across geographically distributed data in this setting. However, an important class of applications have not been explored for multicluster Map Reduce: iterative computation. In this paper, we propose Fed Loop, a composite system aimed at providing iterative Map Reduce computation for geographically distributed data in multicluster settings. Fed Loop is capable of transparently executing both iterative and non-iterative Map Reduce jobs on either a single cluster or multiple clusters. For our performance evaluation, two well-known iterative algorithms was executed over 4 independent clusters (16 physical nodes in total) using Fed Loop: K-Means and Page Rank. Results helped us discover how different iterative applications may differ in execution efficiency for mutlicluster environments and how iterative multicluster computation systems like Fed Loop can be optimized.

Original languageEnglish
Title of host publicationProceedings - 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages755-762
Number of pages8
ISBN (Electronic)9781479965137
DOIs
Publication statusPublished - 2015 Jan 15
Event13th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014 - Beijing, China
Duration: 2014 Sep 242014 Sep 26

Publication series

NameProceedings - 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014

Other

Other13th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014
CountryChina
CityBeijing
Period14-09-2414-09-26

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Safety, Risk, Reliability and Quality
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'FedLoop: Looping on federated MapReduce'. Together they form a unique fingerprint.

  • Cite this

    Wang, C. Y., Tai, T. L., Huang, K. C., Liu, T. E., Chang, J. B., & Shieh, C. K. (2015). FedLoop: Looping on federated MapReduce. In Proceedings - 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014 (pp. 755-762). [7011323] (Proceedings - 2014 IEEE 13th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom 2014). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/TrustCom.2014.99