Exploiting and evaluating mapreduce for large-scale graph mining

Hung Che Lai, Cheng-Te Li, Yi Chen Lo, Shou De Lin

研究成果: Conference contribution

1 引文 (Scopus)

摘要

Graph mining is a popular tcchniquc for discovering the hidden structures or important instances in a graph, but the computational efficiency is usually a cause for concern when dealing with large-scale graphs containing billions of entities. Cloud computing is widely regarded as a feasible solution to the problem. In this work, we present an open source graph mining library called the MapReduce Graph Mining Framework (MGMF) to be a robust and efficient MapReduce-based graph mining tool. We start from dividing graph mining algorithms into four categories and designing a MapReduce framew ork for algorithms in each category. The experimental results show that MGMF is 3 to 20 times more efficient than PEGASUS, a state-of- the-art library for graph mining on MapReduce. Moreover, it provides better coverage of different graph mining algorithms. We also validate our framework on billion-scaled networks to demonstrate that it is scalable to the number of machines. Furthermore, we test and compare the feasibility between single machine and the cloud computing technique. The effects of different file input formats for MapReduce arc investigated as well. Our implemented open-source library can be downloaded from http://mslab.csie.ntu.edu.tw/-noahsark/MGMF/.

原文English
主出版物標題Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012
頁面434-441
頁數8
DOIs
出版狀態Published - 2012 十二月 1
事件2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012 - Istanbul, Turkey
持續時間: 2012 八月 262012 八月 29

出版系列

名字Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012

Other

Other2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012
國家Turkey
城市Istanbul
期間12-08-2612-08-29

指紋

Cloud computing
Computational efficiency

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Software

引用此文

Lai, H. C., Li, C-T., Lo, Y. C., & Lin, S. D. (2012). Exploiting and evaluating mapreduce for large-scale graph mining. 於 Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012 (頁 434-441). [6425727] (Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012). https://doi.org/10.1109/ASONAM.2012.77
Lai, Hung Che ; Li, Cheng-Te ; Lo, Yi Chen ; Lin, Shou De. / Exploiting and evaluating mapreduce for large-scale graph mining. Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012. 2012. 頁 434-441 (Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012).
@inproceedings{ab04784d82c043b1b95c14b83c6a0371,
title = "Exploiting and evaluating mapreduce for large-scale graph mining",
abstract = "Graph mining is a popular tcchniquc for discovering the hidden structures or important instances in a graph, but the computational efficiency is usually a cause for concern when dealing with large-scale graphs containing billions of entities. Cloud computing is widely regarded as a feasible solution to the problem. In this work, we present an open source graph mining library called the MapReduce Graph Mining Framework (MGMF) to be a robust and efficient MapReduce-based graph mining tool. We start from dividing graph mining algorithms into four categories and designing a MapReduce framew ork for algorithms in each category. The experimental results show that MGMF is 3 to 20 times more efficient than PEGASUS, a state-of- the-art library for graph mining on MapReduce. Moreover, it provides better coverage of different graph mining algorithms. We also validate our framework on billion-scaled networks to demonstrate that it is scalable to the number of machines. Furthermore, we test and compare the feasibility between single machine and the cloud computing technique. The effects of different file input formats for MapReduce arc investigated as well. Our implemented open-source library can be downloaded from http://mslab.csie.ntu.edu.tw/-noahsark/MGMF/.",
author = "Lai, {Hung Che} and Cheng-Te Li and Lo, {Yi Chen} and Lin, {Shou De}",
year = "2012",
month = "12",
day = "1",
doi = "10.1109/ASONAM.2012.77",
language = "English",
isbn = "9780769547992",
series = "Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012",
pages = "434--441",
booktitle = "Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012",

}

Lai, HC, Li, C-T, Lo, YC & Lin, SD 2012, Exploiting and evaluating mapreduce for large-scale graph mining. 於 Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012., 6425727, Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012, 頁 434-441, 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012, Istanbul, Turkey, 12-08-26. https://doi.org/10.1109/ASONAM.2012.77

Exploiting and evaluating mapreduce for large-scale graph mining. / Lai, Hung Che; Li, Cheng-Te; Lo, Yi Chen; Lin, Shou De.

Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012. 2012. p. 434-441 6425727 (Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012).

研究成果: Conference contribution

TY - GEN

T1 - Exploiting and evaluating mapreduce for large-scale graph mining

AU - Lai, Hung Che

AU - Li, Cheng-Te

AU - Lo, Yi Chen

AU - Lin, Shou De

PY - 2012/12/1

Y1 - 2012/12/1

N2 - Graph mining is a popular tcchniquc for discovering the hidden structures or important instances in a graph, but the computational efficiency is usually a cause for concern when dealing with large-scale graphs containing billions of entities. Cloud computing is widely regarded as a feasible solution to the problem. In this work, we present an open source graph mining library called the MapReduce Graph Mining Framework (MGMF) to be a robust and efficient MapReduce-based graph mining tool. We start from dividing graph mining algorithms into four categories and designing a MapReduce framew ork for algorithms in each category. The experimental results show that MGMF is 3 to 20 times more efficient than PEGASUS, a state-of- the-art library for graph mining on MapReduce. Moreover, it provides better coverage of different graph mining algorithms. We also validate our framework on billion-scaled networks to demonstrate that it is scalable to the number of machines. Furthermore, we test and compare the feasibility between single machine and the cloud computing technique. The effects of different file input formats for MapReduce arc investigated as well. Our implemented open-source library can be downloaded from http://mslab.csie.ntu.edu.tw/-noahsark/MGMF/.

AB - Graph mining is a popular tcchniquc for discovering the hidden structures or important instances in a graph, but the computational efficiency is usually a cause for concern when dealing with large-scale graphs containing billions of entities. Cloud computing is widely regarded as a feasible solution to the problem. In this work, we present an open source graph mining library called the MapReduce Graph Mining Framework (MGMF) to be a robust and efficient MapReduce-based graph mining tool. We start from dividing graph mining algorithms into four categories and designing a MapReduce framew ork for algorithms in each category. The experimental results show that MGMF is 3 to 20 times more efficient than PEGASUS, a state-of- the-art library for graph mining on MapReduce. Moreover, it provides better coverage of different graph mining algorithms. We also validate our framework on billion-scaled networks to demonstrate that it is scalable to the number of machines. Furthermore, we test and compare the feasibility between single machine and the cloud computing technique. The effects of different file input formats for MapReduce arc investigated as well. Our implemented open-source library can be downloaded from http://mslab.csie.ntu.edu.tw/-noahsark/MGMF/.

UR - http://www.scopus.com/inward/record.url?scp=84874253842&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84874253842&partnerID=8YFLogxK

U2 - 10.1109/ASONAM.2012.77

DO - 10.1109/ASONAM.2012.77

M3 - Conference contribution

AN - SCOPUS:84874253842

SN - 9780769547992

T3 - Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012

SP - 434

EP - 441

BT - Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012

ER -

Lai HC, Li C-T, Lo YC, Lin SD. Exploiting and evaluating mapreduce for large-scale graph mining. 於 Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012. 2012. p. 434-441. 6425727. (Proceedings of the 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2012). https://doi.org/10.1109/ASONAM.2012.77