Workload Alleviation Scheduling Framework to Alleviate Negative Performance Impact of Intermediate Data Skew in Small-Scale MapReduce Cloud

Tzu Chi Huang, Kuo Chih Chu, Jia Huei Lin, Guo Hao Huang, Ce-Kuen Shieh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A MapReduce cloud becomes the essential platform in the cloud computing infrastructure today. Because applications may process input data with different algorithms and logics to produce intermediate data, a MapReduce cloud may suffer intermediate data skew by unevenly distributing intermediate data among nodes at run time. When intermediate data skew happens, a MapReduce cloud not only idles nodes to waste computation resources but also prolongs the application execution progress to hurt user experiences in cloud computing. Instead of the existing solutions that assume many available idle nodes and use computation resources in a loose way, a MapReduce cloud can use the Workload Alleviation Scheduling Framework (W ASF) proposed in this paper to alleviate the negative performance impact of intermediate data skew in a small-scale MapReduce cloud by smartly utilizing computation resources. Besides, a MapReduce cloud is verified with popular applications in experiments to have the outstanding performance improvement with W ASF when intermediate data skew happens.

Original languageEnglish
Title of host publication2018 International Conference on System Science and Engineering, ICSSE 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781538662854
DOIs
Publication statusPublished - 2018 Nov 1
Event2018 International Conference on System Science and Engineering, ICSSE 2018 - New Taipei City, Taiwan
Duration: 2018 Jun 282018 Jun 30

Publication series

Name2018 International Conference on System Science and Engineering, ICSSE 2018

Other

Other2018 International Conference on System Science and Engineering, ICSSE 2018
CountryTaiwan
CityNew Taipei City
Period18-06-2818-06-30

Fingerprint

MapReduce
Skew
Workload
Scheduling
Cloud computing
Cloud Computing
Resources
Vertex of a graph
User Experience
Framework
Experiments
Infrastructure
Logic
Experiment

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Control and Optimization

Cite this

Huang, T. C., Chu, K. C., Lin, J. H., Huang, G. H., & Shieh, C-K. (2018). Workload Alleviation Scheduling Framework to Alleviate Negative Performance Impact of Intermediate Data Skew in Small-Scale MapReduce Cloud. In 2018 International Conference on System Science and Engineering, ICSSE 2018 [8520003] (2018 International Conference on System Science and Engineering, ICSSE 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICSSE.2018.8520003
Huang, Tzu Chi ; Chu, Kuo Chih ; Lin, Jia Huei ; Huang, Guo Hao ; Shieh, Ce-Kuen. / Workload Alleviation Scheduling Framework to Alleviate Negative Performance Impact of Intermediate Data Skew in Small-Scale MapReduce Cloud. 2018 International Conference on System Science and Engineering, ICSSE 2018. Institute of Electrical and Electronics Engineers Inc., 2018. (2018 International Conference on System Science and Engineering, ICSSE 2018).
@inproceedings{15d6663796874bd4b4975fbcc69ddd9d,
title = "Workload Alleviation Scheduling Framework to Alleviate Negative Performance Impact of Intermediate Data Skew in Small-Scale MapReduce Cloud",
abstract = "A MapReduce cloud becomes the essential platform in the cloud computing infrastructure today. Because applications may process input data with different algorithms and logics to produce intermediate data, a MapReduce cloud may suffer intermediate data skew by unevenly distributing intermediate data among nodes at run time. When intermediate data skew happens, a MapReduce cloud not only idles nodes to waste computation resources but also prolongs the application execution progress to hurt user experiences in cloud computing. Instead of the existing solutions that assume many available idle nodes and use computation resources in a loose way, a MapReduce cloud can use the Workload Alleviation Scheduling Framework (W ASF) proposed in this paper to alleviate the negative performance impact of intermediate data skew in a small-scale MapReduce cloud by smartly utilizing computation resources. Besides, a MapReduce cloud is verified with popular applications in experiments to have the outstanding performance improvement with W ASF when intermediate data skew happens.",
author = "Huang, {Tzu Chi} and Chu, {Kuo Chih} and Lin, {Jia Huei} and Huang, {Guo Hao} and Ce-Kuen Shieh",
year = "2018",
month = "11",
day = "1",
doi = "10.1109/ICSSE.2018.8520003",
language = "English",
series = "2018 International Conference on System Science and Engineering, ICSSE 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
booktitle = "2018 International Conference on System Science and Engineering, ICSSE 2018",
address = "United States",

}

Huang, TC, Chu, KC, Lin, JH, Huang, GH & Shieh, C-K 2018, Workload Alleviation Scheduling Framework to Alleviate Negative Performance Impact of Intermediate Data Skew in Small-Scale MapReduce Cloud. in 2018 International Conference on System Science and Engineering, ICSSE 2018., 8520003, 2018 International Conference on System Science and Engineering, ICSSE 2018, Institute of Electrical and Electronics Engineers Inc., 2018 International Conference on System Science and Engineering, ICSSE 2018, New Taipei City, Taiwan, 18-06-28. https://doi.org/10.1109/ICSSE.2018.8520003

Workload Alleviation Scheduling Framework to Alleviate Negative Performance Impact of Intermediate Data Skew in Small-Scale MapReduce Cloud. / Huang, Tzu Chi; Chu, Kuo Chih; Lin, Jia Huei; Huang, Guo Hao; Shieh, Ce-Kuen.

2018 International Conference on System Science and Engineering, ICSSE 2018. Institute of Electrical and Electronics Engineers Inc., 2018. 8520003 (2018 International Conference on System Science and Engineering, ICSSE 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Workload Alleviation Scheduling Framework to Alleviate Negative Performance Impact of Intermediate Data Skew in Small-Scale MapReduce Cloud

AU - Huang, Tzu Chi

AU - Chu, Kuo Chih

AU - Lin, Jia Huei

AU - Huang, Guo Hao

AU - Shieh, Ce-Kuen

PY - 2018/11/1

Y1 - 2018/11/1

N2 - A MapReduce cloud becomes the essential platform in the cloud computing infrastructure today. Because applications may process input data with different algorithms and logics to produce intermediate data, a MapReduce cloud may suffer intermediate data skew by unevenly distributing intermediate data among nodes at run time. When intermediate data skew happens, a MapReduce cloud not only idles nodes to waste computation resources but also prolongs the application execution progress to hurt user experiences in cloud computing. Instead of the existing solutions that assume many available idle nodes and use computation resources in a loose way, a MapReduce cloud can use the Workload Alleviation Scheduling Framework (W ASF) proposed in this paper to alleviate the negative performance impact of intermediate data skew in a small-scale MapReduce cloud by smartly utilizing computation resources. Besides, a MapReduce cloud is verified with popular applications in experiments to have the outstanding performance improvement with W ASF when intermediate data skew happens.

AB - A MapReduce cloud becomes the essential platform in the cloud computing infrastructure today. Because applications may process input data with different algorithms and logics to produce intermediate data, a MapReduce cloud may suffer intermediate data skew by unevenly distributing intermediate data among nodes at run time. When intermediate data skew happens, a MapReduce cloud not only idles nodes to waste computation resources but also prolongs the application execution progress to hurt user experiences in cloud computing. Instead of the existing solutions that assume many available idle nodes and use computation resources in a loose way, a MapReduce cloud can use the Workload Alleviation Scheduling Framework (W ASF) proposed in this paper to alleviate the negative performance impact of intermediate data skew in a small-scale MapReduce cloud by smartly utilizing computation resources. Besides, a MapReduce cloud is verified with popular applications in experiments to have the outstanding performance improvement with W ASF when intermediate data skew happens.

UR - http://www.scopus.com/inward/record.url?scp=85057622088&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057622088&partnerID=8YFLogxK

U2 - 10.1109/ICSSE.2018.8520003

DO - 10.1109/ICSSE.2018.8520003

M3 - Conference contribution

AN - SCOPUS:85057622088

T3 - 2018 International Conference on System Science and Engineering, ICSSE 2018

BT - 2018 International Conference on System Science and Engineering, ICSSE 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Huang TC, Chu KC, Lin JH, Huang GH, Shieh C-K. Workload Alleviation Scheduling Framework to Alleviate Negative Performance Impact of Intermediate Data Skew in Small-Scale MapReduce Cloud. In 2018 International Conference on System Science and Engineering, ICSSE 2018. Institute of Electrical and Electronics Engineers Inc. 2018. 8520003. (2018 International Conference on System Science and Engineering, ICSSE 2018). https://doi.org/10.1109/ICSSE.2018.8520003