An inter-framework cache for diverse data-intensive computing environments

Chun Yu Wang, Tzu En Huang, Yu Tang Huang, Jyh Biau Chang, Ce Kuen Shieh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Hadoop Distributed File System (HDFS) provides the storage to keep analyzing outcomes for the diversity of frameworks. MapReduce, Storm, and Spark each applies on batching, streaming and in-memory computing, all of them need the HDFS to collect and assemble results. For coping with Big-Data analysis in the real world, complicated platforms required working together. However, collaborating analysis on heterogeneous frameworks, the data must be write-through firstly and post-fetch upon HDFS that degrades the performance and lower the effectiveness of the whole system. For best our knowledge, no previous work had focused on inter-framework data caching. To solve above problems on collaborating analysis within heterogeneous frameworks such as Hadoop and Strom, in this paper, we propose a cache system upon YARN called "Inter-Framework Cache" (IF-cache). It uses in-memory cache to reserve temporary outcomes while also reducing the HDFS access frequency and improve analysis performance. Experiments had shown that Hadoop with IF-cache can reduce about 50% times comparing the no-cache one.

Original languageEnglish
Title of host publicationProceedings - 2015 IEEE International Conference on Smart City, SmartCity 2015, Held Jointly with 8th IEEE International Conference on Social Computing and Networking, SocialCom 2015, 5th IEEE International Conference on Sustainable Computing and Communications, SustainCom 2015, 2015 International Conference on Big Data Intelligence and Computing, DataCom 2015, 5th International Symposium on Cloud and Service Computing, SC2 2015
EditorsXingang Liu, Peicheng Wang, Yufeng Wang, Mianxiong Dong, Robert C. H. Hsu, Feng Xia, Yuhui Deng
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages944-949
Number of pages6
ISBN (Electronic)9781509018932
DOIs
Publication statusPublished - 2015 Jan 1
EventIEEE International Conference on Smart City, SmartCity 2015 - Chengdu, China
Duration: 2015 Dec 192015 Dec 21

Publication series

NameProceedings - 2015 IEEE International Conference on Smart City, SmartCity 2015, Held Jointly with 8th IEEE International Conference on Social Computing and Networking, SocialCom 2015, 5th IEEE International Conference on Sustainable Computing and Communications, SustainCom 2015, 2015 International Conference on Big Data Intelligence and Computing, DataCom 2015, 5th International Symposium on Cloud and Service Computing, SC2 2015

Other

OtherIEEE International Conference on Smart City, SmartCity 2015
CountryChina
CityChengdu
Period15-12-1915-12-21

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Media Technology
  • Computer Science Applications
  • Signal Processing
  • Computer Networks and Communications
  • Modelling and Simulation
  • Sociology and Political Science
  • Urban Studies

Fingerprint Dive into the research topics of 'An inter-framework cache for diverse data-intensive computing environments'. Together they form a unique fingerprint.

Cite this