Speaker: Xiaolan (Catherine) Zhang, Computer Science, Harvard University
Application-Specific Benchmarking
Abstract: In this talk I will introduce a novel approach to performance evaluation, called application-specific benchmarking, and present techniques for designing and constructing meaningful benchmarks.
Conventional benchmarks (such as SPEC) assume a workload consisting of a certain mix of programs. They are very useful in predicting the performance of applications that exhibit similar behaviors to those of the benchmark programs. However, if the intended application does not behave the same way as the benchmark programs, the benchmark scores are often uninformative, and sometimes can be misleading.
The application-specific benchmarking approach incorporates characteristics of the application of interest into the benchmarking process, yielding performance metrics that reflect the expected behavior of a particular application across a range of different platforms. It also allows benchmarks to evolve with applications and consequently the benchmarks are always up-to-date. In this talk I will present this methodology and describe how we applied it to evaluating the performance of Java Virtual Machines (HBench:Java) and garbage collectors (HBench:JGC) in general. Our experimental results demonstrate that HBench:Java is able to correctly predict the rank of running times of three commercial applications on a variety of Java Virtual Machine implementations. In the realm of garbage collection, the predicted garbage collection times for a stop-the-world, mark-sweep garbage collector closely match the actual times.
At the end of my talk, I will also briefly discuss two other systems
projects that I carried out before: 1) HACC, a clustered web-server
architecture that achieves high performance by improving the total cache
efficiency of the cluster; and 2) Morph, a framework for continuous profiling
and automatic optimization to attain better performance by adapting applications
to individual usages and/or specific machine configurations.
Short Bio: Xiaolan
(Catherine) Zhang is a Ph.D candidate in Computer Science at
the Division of Engineering and Applied Sciences of Harvard University.
Catherine graduated with a B.S. from Shanghai Jiao Tong University, P.R.China
in 1993, and a M.S. from Northeastern University in 1995, both in Computer
Science. She has worked at GTE Laboratories, Digital's System Research
Center (SRC) and Sun Microsystem Labs East during past summers. Her
thesis research concerns novel approaches to system performance evaluation.
She has also worked on a few projects covering a broad range of systems
topics, including profiling, compiler optimization, and high-performance
web server architectures..
Hosts: C.
Healey, Computer Science, NCSU