Bridging the Gap between Big Data System Software Stack and Applications: The Case of Semiconductor Wafer Fabrication Foundries

Chia Ping Tsai, Hung Chang Hsiao, Yu Chang Chao, Michael Hsu, Andy R.K. Chang

研究成果: Conference contribution

摘要

We present in this paper two novel infrastructural services based on Hadoop for big data storage and computing in a Taiwan's semiconductor wafer fabrication foundry. The two services include Hadoop data service (HDS) and distributed R language computing service (DRS), which have been built and operated in production systems for 3.5 years. They evolve over time by incrementally accommodating users' requirements. HDS is a web- based distributed big data storage facility. Users simply rely on HDS to access data objects stored in Hadoop with the HTTP protocol. In addition, HDS is scalable and reliable. Moreover, HDS is efficient and effective by intelligently selecting either Hadoop distributed file system (HDFS) or database (HBase) for publishing data objects. Specifically, HDS is transparent to existing analytics and data inquiry applications, such as Spark and Hive. While HDS is a unified storage for supporting sequential and random data accesses for big data in the wafer fabrication foundry, DRS is a distributed computing framework for typical R language users. R users employ DRS to enjoy data-parallel computations, effortlessly and seamlessly. Similar to HDS, DRS can be horizontally scaled out. It guarantees the completion of computational jobs even with failures. In particular, it adaptively reallocates computational resources on the fly, minimizing job execution time and maximizing utilization of allocated resources. This paper discusses the design and implementation features for HDS and DRS. It also demonstrates their performance metrics.

原文English
主出版物標題Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018
編輯Yang Song, Bing Liu, Kisung Lee, Naoki Abe, Calton Pu, Mu Qiao, Nesreen Ahmed, Donald Kossmann, Jeffrey Saltz, Jiliang Tang, Jingrui He, Huan Liu, Xiaohua Hu
發行者Institute of Electrical and Electronics Engineers Inc.
頁面1865-1874
頁數10
ISBN(電子)9781538650356
DOIs
出版狀態Published - 2019 一月 22
事件2018 IEEE International Conference on Big Data, Big Data 2018 - Seattle, United States
持續時間: 2018 十二月 102018 十二月 13

出版系列

名字Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

Conference

Conference2018 IEEE International Conference on Big Data, Big Data 2018
國家United States
城市Seattle
期間18-12-1018-12-13

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Information Systems

指紋 深入研究「Bridging the Gap between Big Data System Software Stack and Applications: The Case of Semiconductor Wafer Fabrication Foundries」主題。共同形成了獨特的指紋。

  • 引用此

    Tsai, C. P., Hsiao, H. C., Chao, Y. C., Hsu, M., & Chang, A. R. K. (2019). Bridging the Gap between Big Data System Software Stack and Applications: The Case of Semiconductor Wafer Fabrication Foundries. 於 Y. Song, B. Liu, K. Lee, N. Abe, C. Pu, M. Qiao, N. Ahmed, D. Kossmann, J. Saltz, J. Tang, J. He, H. Liu, & X. Hu (編輯), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018 (頁 1865-1874). [8621954] (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData.2018.8621954