8. Benchmarking Java Applications

It is a common practice to evaluate the performance of an application or make inferences about the performance of an application through the use of benchmarks. Benchmarks are programs specifically developed to measure the performance of one or more elements of a computing system. In the context of Java software, benchmarks are Java programs intended to measure the performance of one or more elements of a system where the Java program is being executed. These elements can include the entire hardware and software stack or be limited to a small segment of functionality in a Java program. The latter is often described as a micro-benchmark since it has a much narrower focus. Benchmarks used to make the broader evaluation of a system’s performance, that is, the entire hardware and software stack, are often industry standard benchmarks such as those developed under the collaboration of industry competitors such as those developed at SPEC (Standard Performance Evaluation Corporation). In contrast, micro-benchmarks tend to be created by developers since micro-benchmarks have a narrow or specific performance question to be analyzed.

Developing benchmarks, especially micro-benchmarks, to execute in a modern Java Virtual Machine (JVM) introduces numerous challenges that can often lead an observer and a developer of a benchmark to improper or incorrect conclusions due to one or more of the many runtime optimizations that can be made by modern JVMs such as the Java HotSpot VM (also referred to as HotSpot VM hereafter). The development of Java benchmarks, including micro-benchmarks, is largely an art. This chapter describes several potential issues to be aware of when writing Java benchmarks, including micro-benchmarks, and how to identify potential issues with a micro-benchmark along with tips to effectively write Java benchmarks. Additionally, how to identify performance improvements or regressions with benchmarks through the use of the design of experiments and statistical methods to improve confidence in arriving at conclusions are also presented. To gain the most from this chapter, it may be helpful to read or review the “HotSpot VM JIT Compilers” section of Chapter 3, “JVM Overview.” Doing so will help you learn about some of the complex optimizations a modern JVM’s JIT compiler can perform on a Java application.