Virtual Hadoop: MapReduce over docker containers with an auto-scaling mechanism for heterogeneous environments

Yi Wei Chen, Shih Hao Hung, ChiaHeng Tu, Chih Wei Yeh

研究成果: Conference contribution

5 引文 斯高帕斯(Scopus)

摘要

Hadoop is a widely used software framework for handling massive data. As heterogeneous computing gains its momentum, variants of Hadoop have been developed to Offoad the computation of the Hadoop applications onto the heterogeneous processors, such as GPUS, DSPs, and FPGA. Unfortunately, these variants do not support on-demand resource scaling for the deadline-aware applications in a sophisticated heterogeneous computing environment. In this work, we developed a framework called Virtual Hadoop, which scales out the required computing resources for the applications automatically to meet the given real-time requirements. We extended the methods of resource inference and allocation for the heterogeneous computing environments. On top of these methods, an auto-scaling mechanism was developed to dynamically allocate resources on-demand based on profile data and performance models for the application execution to meet the given time requirements. In addition, Virtual Hadoop can utilize Docker containers to facilitate the auto-scaling mechanism, where a container encapsulates a Hadoop node with the capability to leverage heterogeneous computing engines. Our experimental results reveal the efficiency of Virtual Hadoop, and hopefully the experiences and discussion presented in this paper will ease the adoption of heterogeneous computing for Efficient big data processing. Copyright is held by the owner/author(s). Publication rights licensed to ACM.

原文English
主出版物標題Proceedings of the 2016 Research in Adaptive and Convergent Systems, RACS 2016
發行者Association for Computing Machinery, Inc
頁面201-206
頁數6
ISBN(電子)9781450344555
DOIs
出版狀態Published - 2016 十月 11
事件2016 Research in Adaptive and Convergent Systems, RACS 2016 - Odense, Denmark
持續時間: 2016 十月 112016 十月 14

出版系列

名字Proceedings of the 2016 Research in Adaptive and Convergent Systems, RACS 2016

Other

Other2016 Research in Adaptive and Convergent Systems, RACS 2016
國家Denmark
城市Odense
期間16-10-1116-10-14

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Computer Science(all)

指紋 深入研究「Virtual Hadoop: MapReduce over docker containers with an auto-scaling mechanism for heterogeneous environments」主題。共同形成了獨特的指紋。

  • 引用此

    Chen, Y. W., Hung, S. H., Tu, C., & Yeh, C. W. (2016). Virtual Hadoop: MapReduce over docker containers with an auto-scaling mechanism for heterogeneous environments. 於 Proceedings of the 2016 Research in Adaptive and Convergent Systems, RACS 2016 (頁 201-206). (Proceedings of the 2016 Research in Adaptive and Convergent Systems, RACS 2016). Association for Computing Machinery, Inc. https://doi.org/10.1145/2987386.2987408