An inter-framework cache for diverse data-intensive computing environments

Chun Yu Wang, Tzu En Huang, Yu Tang Huang, Jyh Biau Chang, Ce Kuen Shieh

研究成果: Conference contribution

摘要

Hadoop Distributed File System (HDFS) provides the storage to keep analyzing outcomes for the diversity of frameworks. MapReduce, Storm, and Spark each applies on batching, streaming and in-memory computing, all of them need the HDFS to collect and assemble results. For coping with Big-Data analysis in the real world, complicated platforms required working together. However, collaborating analysis on heterogeneous frameworks, the data must be write-through firstly and post-fetch upon HDFS that degrades the performance and lower the effectiveness of the whole system. For best our knowledge, no previous work had focused on inter-framework data caching. To solve above problems on collaborating analysis within heterogeneous frameworks such as Hadoop and Strom, in this paper, we propose a cache system upon YARN called "Inter-Framework Cache" (IF-cache). It uses in-memory cache to reserve temporary outcomes while also reducing the HDFS access frequency and improve analysis performance. Experiments had shown that Hadoop with IF-cache can reduce about 50% times comparing the no-cache one.

原文English
主出版物標題Proceedings - 2015 IEEE International Conference on Smart City, SmartCity 2015, Held Jointly with 8th IEEE International Conference on Social Computing and Networking, SocialCom 2015, 5th IEEE International Conference on Sustainable Computing and Communications, SustainCom 2015, 2015 International Conference on Big Data Intelligence and Computing, DataCom 2015, 5th International Symposium on Cloud and Service Computing, SC2 2015
編輯Xingang Liu, Peicheng Wang, Yufeng Wang, Mianxiong Dong, Robert C. H. Hsu, Feng Xia, Yuhui Deng
發行者Institute of Electrical and Electronics Engineers Inc.
頁面944-949
頁數6
ISBN(電子)9781509018932
DOIs
出版狀態Published - 2015 一月 1
事件IEEE International Conference on Smart City, SmartCity 2015 - Chengdu, China
持續時間: 2015 十二月 192015 十二月 21

出版系列

名字Proceedings - 2015 IEEE International Conference on Smart City, SmartCity 2015, Held Jointly with 8th IEEE International Conference on Social Computing and Networking, SocialCom 2015, 5th IEEE International Conference on Sustainable Computing and Communications, SustainCom 2015, 2015 International Conference on Big Data Intelligence and Computing, DataCom 2015, 5th International Symposium on Cloud and Service Computing, SC2 2015

Other

OtherIEEE International Conference on Smart City, SmartCity 2015
國家China
城市Chengdu
期間15-12-1915-12-21

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Media Technology
  • Computer Science Applications
  • Signal Processing
  • Computer Networks and Communications
  • Modelling and Simulation
  • Sociology and Political Science
  • Urban Studies

指紋 深入研究「An inter-framework cache for diverse data-intensive computing environments」主題。共同形成了獨特的指紋。

  • 引用此

    Wang, C. Y., Huang, T. E., Huang, Y. T., Chang, J. B., & Shieh, C. K. (2015). An inter-framework cache for diverse data-intensive computing environments. 於 X. Liu, P. Wang, Y. Wang, M. Dong, R. C. H. Hsu, F. Xia, & Y. Deng (編輯), Proceedings - 2015 IEEE International Conference on Smart City, SmartCity 2015, Held Jointly with 8th IEEE International Conference on Social Computing and Networking, SocialCom 2015, 5th IEEE International Conference on Sustainable Computing and Communications, SustainCom 2015, 2015 International Conference on Big Data Intelligence and Computing, DataCom 2015, 5th International Symposium on Cloud and Service Computing, SC2 2015 (頁 944-949). [7463847] (Proceedings - 2015 IEEE International Conference on Smart City, SmartCity 2015, Held Jointly with 8th IEEE International Conference on Social Computing and Networking, SocialCom 2015, 5th IEEE International Conference on Sustainable Computing and Communications, SustainCom 2015, 2015 International Conference on Big Data Intelligence and Computing, DataCom 2015, 5th International Symposium on Cloud and Service Computing, SC2 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SmartCity.2015.192