Smart partitioning mechanism for dealing with intermediate data skew in reduce task on cloud computing

Tzu Chi Huang, Kuo Chih Chu, Guo Hao Huang, Yan Chen Shen, Ce Kuen Shieh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

MapReduce greatly alleviates the burdens of programmers and gradually becomes an application programming standard on cloud computing nowadays, because the run-time system of cloud computing can automatically handle the issues of paralleled and distributed programming on behalf of programmers at run time. Although MapReduce can strongly benefit programmers on developing cloud computing applications, intermediate data skew inevitably hurts application performances. MapReduce can use the Smart Partitioning Mechanism (SPM) proposed in this paper as an alternative solution to deal with intermediate data skew in Reduce tasks on cloud computing. With the capability of averagely distributing intermediate data over Slave nodes in SPM, MapReduce no longer suffers from the performance penalty resulting from the intermediate data skew problem in Reduce tasks on cloud computing.

Original languageEnglish
Title of host publicationProceedings - 31st IEEE International Conference on Advanced Information Networking and Applications, AINA 2017
EditorsTomoya Enokido, Hui-Huang Hsu, Chi-Yi Lin, Makoto Takizawa, Leonard Barolli
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages819-826
Number of pages8
ISBN (Electronic)9781509060283
DOIs
Publication statusPublished - 2017 May 5
Event31st IEEE International Conference on Advanced Information Networking and Applications, AINA 2017 - Taipei, Taiwan
Duration: 2017 Mar 272017 Mar 29

Publication series

NameProceedings - International Conference on Advanced Information Networking and Applications, AINA
ISSN (Print)1550-445X

Other

Other31st IEEE International Conference on Advanced Information Networking and Applications, AINA 2017
CountryTaiwan
CityTaipei
Period17-03-2717-03-29

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Fingerprint Dive into the research topics of 'Smart partitioning mechanism for dealing with intermediate data skew in reduce task on cloud computing'. Together they form a unique fingerprint.

  • Cite this

    Huang, T. C., Chu, K. C., Huang, G. H., Shen, Y. C., & Shieh, C. K. (2017). Smart partitioning mechanism for dealing with intermediate data skew in reduce task on cloud computing. In T. Enokido, H-H. Hsu, C-Y. Lin, M. Takizawa, & L. Barolli (Eds.), Proceedings - 31st IEEE International Conference on Advanced Information Networking and Applications, AINA 2017 (pp. 819-826). [7920992] (Proceedings - International Conference on Advanced Information Networking and Applications, AINA). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/AINA.2017.44