Dynamic Grouping integrated Neighboring Search Job Allocation Scheduler for Hadoop MapReduce in Heterogeneous Computing Environments

  • 陳 紀廷

Student thesis: Master's Thesis


With the rapid development of the Internet the growth of the cloud environment and the amount of explosive increasing network data cloud computing which was a new noun in distributed computing systems in recent years become a hot term MapReduce is one of a very important cloud computing architecture while the Apache Hadoop is one of the more well-known implement MapReduce and cloud computing platforms The resources required for jobs executed in a large data center very according to the type of jobs Gen-erally there has two kinds of Jobs CPU-bound jobs and I/O-bound jobs which demand di?erent resources but run simultaneously in the same cluster The default job scheduler of Hadoop is ?rst-come-?rst-served (FCFS) and thus may cause unbalance resource uti-lization Given various job workloads the JAS categorizes jobs and then assigns tasks to a CPU-bound queue or an I/O-bound queue However the JAS exhibited a locality problem which was addressed by developing a modi?ed JAS called the job allocation scheduler with locality (JASL) and create dynamic job allocation scheduler with local-ity (DJASL) which exhibited better performance and reduce extra network tra?c ?ow But the drawback of (DJASL) is (DJASL) e?ectively enhance data locality but failed to have signi?cant growth on job execution performance Therefore in this paper we proposes a job scheduler with dynamic grouping integrated neighboring search strategy called (DGNS) which designed to balance resource utilization and take performance and data locality improvement into account in heterogeneous computing environments The DGNS algorithm exhibits more favorable performance and data locality compared with Hadoop DMR JAS and DJASL
Date of Award2015 Aug 31
Original languageEnglish
SupervisorSun-Yuan Hsieh (Supervisor)

Cite this