A data compacting technique to reduce the NetFlow size in botnet detection with BotCluster

Chun Yu Wang, Yu Cheng Chen, Shih Hao Fuh, Feng Min Cho, Ta Chun Lo, Jyh Biau Chang, Qi Jun Cheng, Ce Kuen Shieh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Big data analytics helps us to find potentially valuable knowledge, but as the size of the dataset increases, the computing cost also grows exponentially. In our previous work, BotCluster, we had designed a pre-processing filtering pipeline, including whitelist filter and flow loss-response rate (FLR) filter, for data reduction, which intended to wipe out irrelative noises and reduce the computing overhead. However, we still face a data redundancy phenomenon in which some of the same feature vectors repeatedly emerged. In this paper, we propose a data compacting approach aimed to reduce the input volume and keep enough representative feature vectors to fit DBSCAN's (Density-based spatial clustering of applications with noise) criteria. It purges the redundant vectors according to a purging threshold and keeps the primary representatives. Experimental results have shown that the average data reduction ratio is about 81.34%, while the precision has only slightly decreased by 1.6% on average, and the results still have 99.88% of IPs overlapped with the previous system.

Original languageEnglish
Title of host publicationBDCAT 2019 - Proceedings of the 6th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies
PublisherAssociation for Computing Machinery, Inc
Pages81-84
Number of pages4
ISBN (Electronic)9781450370165
DOIs
Publication statusPublished - 2019 Dec 2
Event6th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, BDCAT 2019 - Auckland, New Zealand
Duration: 2019 Dec 22019 Dec 5

Publication series

NameBDCAT 2019 - Proceedings of the 6th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies

Conference

Conference6th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, BDCAT 2019
CountryNew Zealand
CityAuckland
Period19-12-0219-12-05

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Computer Science Applications
  • Information Systems
  • Decision Sciences (miscellaneous)
  • Information Systems and Management
  • Communication

Fingerprint Dive into the research topics of 'A data compacting technique to reduce the NetFlow size in botnet detection with BotCluster'. Together they form a unique fingerprint.

  • Cite this

    Wang, C. Y., Chen, Y. C., Fuh, S. H., Cho, F. M., Lo, T. C., Chang, J. B., Cheng, Q. J., & Shieh, C. K. (2019). A data compacting technique to reduce the NetFlow size in botnet detection with BotCluster. In BDCAT 2019 - Proceedings of the 6th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies (pp. 81-84). (BDCAT 2019 - Proceedings of the 6th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies). Association for Computing Machinery, Inc. https://doi.org/10.1145/3365109.3368778