Load Balancing Algorithms and Their Impacts on Apache Kafka

Hung Chang Hsiao, Chia Ping Tsai, Zheng Xian Li, Chao Heng Lee, Jia Sheng Chen, Yu Chen Lai, Jia Chi Wang, Shao Chi Li, Jhih Cyuan Gao, Yi Huan Lee

研究成果: Conference contribution

摘要

Apache Kafka is a distributed data streaming platform that is widely adopted in the industry. Producers introduce messages to Kafka server nodes, and consumers fetch interested messages in real time. In Kafka, the loads of nodes that serve streaming traffic may be imbalanced because some traffic is relatively popular, resulting in load imbalance. Balancing loads among nodes enables producers and consumers to avoid performance bottleneck, shortening the latencies of sending and receiving their messages and thus improving quality of service. Partitions are the fundamental entities that serve loads in Kafka and are responsible for hosting message payloads. Kafka provides a built-in load balancer that addresses load imbalance. The built-in load balancer distributes partitions evenly to the system, which is not designed to deal with load imbalance due to the heterogeneity of loads. The state-of-the-art load balancer released by LinkedIn, namely, Cruise Control (CC), complements the Kafka built-in load balancer. The performance quality of CC is sensitive and highly depends on the ordering of performance metric constraints. We propose Yet Another (YA) load balancer for Kafka, aiming to achieve simplicity and robustness for the average case. We compare YA with the Kafka built-in load balancer and CC in a real cluster environment. Performance results validate our findings and indicate that load balancers based on gathered performance metrics can effectively reduce the end-to-end delay perceived by streaming applications up to a ratio of approximately 20. CC and our proposed load balancer clearly outperform the Kafka built-in load balancer, yet while our proposed load balancer is comparable to CC, it is more robust.

原文English
主出版物標題Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023
編輯Jingrui He, Themis Palpanas, Xiaohua Hu, Alfredo Cuzzocrea, Dejing Dou, Dominik Slezak, Wei Wang, Aleksandra Gruca, Jerry Chun-Wei Lin, Rakesh Agrawal
發行者Institute of Electrical and Electronics Engineers Inc.
頁面1726-1735
頁數10
ISBN(電子)9798350324457
DOIs
出版狀態Published - 2023
事件2023 IEEE International Conference on Big Data, BigData 2023 - Sorrento, Italy
持續時間: 2023 12月 152023 12月 18

出版系列

名字Proceedings - 2023 IEEE International Conference on Big Data, BigData 2023

Conference

Conference2023 IEEE International Conference on Big Data, BigData 2023
國家/地區Italy
城市Sorrento
期間23-12-1523-12-18

All Science Journal Classification (ASJC) codes

  • 人工智慧
  • 電腦網路與通信
  • 電腦科學應用
  • 資訊系統
  • 資訊系統與管理
  • 安全、風險、可靠性和品質

指紋

深入研究「Load Balancing Algorithms and Their Impacts on Apache Kafka」主題。共同形成了獨特的指紋。

引用此