A systematic methodology for OS benchmark characterization

Using benchmarks to evaluate operating systems is a common and important approach. However, determining which benchmarks to use for such evaluation requires very careful consideration. It has been found that a seemingly naive change of system configuration or input set could lead to drastic change of benchmark characteristics, and could also lead to misleading or incorrect results. Some OS benchmark suites may also include too many benchmark programs with very similar characteristics that could give biased results against, or in favor of, certain kernel behavior. Hence, we need to determine the characteristics of benchmark programs in order to come up with an appropriate benchmark suite for such evaluation, and to interpret the measured results more precisely and correctly. Although there have been many tools developed to help on profiling an OS and to characterize its run-time behavior, the collected data by those tools are often very large and complex. It is extremely time consuming, labor intensive, and error prone to analyze the large volume of measured results, and to determine the characteristics of a suite of benchmark programs. In this work, we propose to use machine-learning techniques to help on analyzing and characterizing OS benchmark programs based on the traced OS kernel events. In this work, a systematic methodology is proposed to automatically characterize benchmarks. We found that the characterized OS behavior could help developers to choose appropriate applications to benchmark operating systems.