A Fast and Scalable Cluster Simulator for Network Performance Projection of HPC Applications

Cheng Yueh Liu, Po Yao Huang, ChiaHeng Tu, Shih Hao Hung

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

For high-performance computing (HPC) cluster to solve distributed computing problems with big data, the performance is usually bounded by communication ability of each node and the underlying network architecture of the cluster. Network simulations are commonly adopted to evaluate the impacts of network configurations before the deployment of such a high-cost system, and a fast and scalable network performance simulation with profiling mechanism is crucial to explore the design space in time effectively. This work presents Clusim, a cluster simulator for exploration of the performance of interplay among distributed HPC applications and network architectures, where three primary techniques are developed in Clusim·. 1) the full system emulators forming a virtual cluster for collecting the network communication traces, 2) window-based simulation, and 3) phase-Aware accelerated network simulation. The first technique helps to reduce the cost and efforts of establishing a physical cluster system for performance estimation. The second one enables cross-platform performance projection by thoroughly preserving the concurrency of the recorded packet flows during simulation. The last one efficiently reduces the simulation time by skipping the detailed simulations of repeated phases of network traffics. We validate Clusim against a physical cluster and show that the difference of the estimated end-To-end delays of the simulated network and those measured on the physical counterpart is within 15%. Meanwhile, thanks to the phase-Aware simulation scheme, Clusim achieves up to 18x speedups, compared with the packet-by-packet simulation by ns-3. Our results suggest that Clusim is a promising design and can rapidly provide the network performance projection of a large scale system with the appropriate trade-off between accuracy and time.

Original languageEnglish
Title of host publicationProceedings - 2018 International Conference on High Performance Computing and Simulation, HPCS 2018
EditorsKhalid Zine-Dine, Waleed W. Smari
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages970-977
Number of pages8
ISBN (Electronic)9781538678787
DOIs
Publication statusPublished - 2018 Oct 29
Event16th International Conference on High Performance Computing and Simulation, HPCS 2018 - Orleans, France
Duration: 2018 Jul 162018 Jul 20

Publication series

NameProceedings - 2018 International Conference on High Performance Computing and Simulation, HPCS 2018

Other

Other16th International Conference on High Performance Computing and Simulation, HPCS 2018
CountryFrance
CityOrleans
Period18-07-1618-07-20

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Computer Networks and Communications
  • Safety, Risk, Reliability and Quality
  • Modelling and Simulation
  • Instrumentation

Fingerprint Dive into the research topics of 'A Fast and Scalable Cluster Simulator for Network Performance Projection of HPC Applications'. Together they form a unique fingerprint.

Cite this