PhD research
Dissertation wrestling
Feedback from one of my advisors.
Statistically Rigorous Java Performance Evaluation
The following paper has also been accepted for publication at OOPSLA 2007: Statistically Rigorous Java Performance Evaluation (PDF, 1.6MB).
Paper abstract
Java performance is far from being trivial to benchmark because it is affected by various factors such as the Java application, its input, the virtual machine, the garbage collector, the heap size, etc. In addition, non-determinism at run-time causes the execution time of a Java program to differ from run to run. There are a number of sources of non-determinism such as Just-In-Time (JIT) compilation and optimization in the virtual machine (VM) driven by timer-based method sampling, thread scheduling, garbage collection, and various system effects.
There exist a wide variety of Java performance evaluation methodologies used by researchers and benchmarkers. These methodologies differ from each other in a number of ways. Some report average performance over a number of runs of the same experiment; others report the best or second best performance observed; yet others report the worst. Some iterate the benchmark multiple times within a single VM invocation; others consider multiple VM invocations and iterate a single benchmark execution; yet others consider multiple VM invocations and iterate the benchmark multiple times.
This paper shows that prevalent methodologies can be misleading, and can even lead to incorrect conclusions. The reason is that the data analysis is not statistically rigorous. In this paper, we present a survey of existing Java performance evaluation methodologies and discuss the importance of statistically rigorous data analysis for dealing with non-determinism. We advocate approaches to quantify startup as well as steady-state performance, and, in addition, we provide the JavaStats software to automatically obtain performance numbers in a rigorous manner. Although this paper focuses on Java performance evaluation, many of the issues addressed in this paper also apply to other programming languages and systems that build on a managed runtime system.
Using HPM-Sampling to Drive Dynamic Compilation
Our paper Using HPM-Sampling to Drive Dynamic Compilation (PDF, 0.6MB) has been accepted for publication at OOPSLA'07.
Good news because OOPSLA, which is short for ACM conference on object-oriented programming, systems, languages, and applications has incubated many state-of-the-art technologies, including design patterns, refactoring, aspect-oriented software development, dynamic compilation and optimization, the Unified Modeling Language, and more.
Paper abstract
All high-performance production Java virtual machines employ an adaptive strategy for program execution. Methods are first executed unoptimized and then an online profiling mechanism is used to find a subset of methods that should be optimized during the same execution. This paper empirically evaluates the design space of several profilers for initiating dynamic compilation and shows that existing online profiling schemes suffer from several limitations. They provide an insufficient number of samples, are untimely, and have limited accuracy at determining the frequently executed methods. We describe and comprehensively evaluate HPM-sampling, a simple but effective profiling scheme for finding optimization candidates using hardware performance monitors (HPMs) that addresses the aforementioned limitations. We show that HPM-sampling is more accurate; has low overhead; and improves performance by 5.7% on average and up to 18.3% when compared to the default system in Jikes RVM, without changing the JIT compiler.
Javana
Our paper Javana: a system for building customized Java program analysis tools (PDF, 1.3MB) has been accepted for publication at OOPSLA'06 (17% acceptance rate).
Good news because OOPSLA, which is short for ACM conference on object-oriented programming, systems, languages, and applications has incubated many state-of-the-art technologies, including design patterns, refactoring, aspect-oriented software development, dynamic compilation and optimization, the Unified Modeling Language, and more.
Paper abstract
Understanding the behavior of applications running on high-level language virtual machines, as is the case in Java, is non-trivial because of the tight entanglement at the lowest execution level between the application and the virtual machine. This paper proposes Javana, a system for building Java program analysis tools. Javana provides an easy-to-use instrumentation infrastructure that allows for building customized profiling tools very quickly.
Javana runs a dynamic binary instrumentation tool underneath the virtual machine. The virtual machine communicates with the instrumentation layer through an event handling mechanism for building a vertical map that links low-level native instruction pointers and memory addresses to high-level language concepts such as objects, methods, threads, lines of code, etc. The dynamic binary instrumentation tool then intercepts all memory accesses and instructions executed and provides the Javana end user with high-level language information for all memory accesses and natively executed instructions.
We demonstrate the power of Javana through a number of applications: memory address tracing, vertical cache simulation and object lifetime computation. For each of these applications, the instrumentation specification requires only a small number of lines of code. Developing similarly powerful profiling tools within a virtual machine (as done in current practice) is both time-consuming and error-prone; in addition, the accuracy of the obtained profiling results might be questionable as we show in this paper.
Garbage collection hints
Our paper GCH: Hints for Triggering Garbage Collections (PDF, 905KB) has been published in Transactions on High-Performance Embedded Architectures and Compilers.
Paper abstract
This paper shows that Appel-style garbage collectors often make suboptimal decisions both in terms of when and how to collect. We argue that garbage collection should be done when the amount of live bytes is low (in order to minimize the collection cost) and when the amount of dead objects is high (in order to maximize the available heap size after collection). In addition, we observe that Appel-style collectors sometimes trigger a nursery collection in cases where a full-heap collection would have been better.
Based on these observations, we propose garbage collection hints (GCH) which is a profile-directed method for guiding garbage collection. Off-line profiling is used to identify favorable collection points in the program code. In those favorable collection points, the garbage collector dynamically chooses between nursery and
full-heap collections based on an analytical garbage collector cost-benefit model. By doing so, GCH guides the collector in terms of when and how to collect.
Experimental results using the SPECjvm98 benchmarks and two generational garbage collectors show that substantial reductions can be obtained in garbage collection time (up to 29X) and that the overall execution time can be reduced by more than 10%. In addition, we also show that GCH reduces the maximum pause times and outperforms
user-inserted forced garbage collections.
Background
The paper is an extended version of the garbage collection hints paper (PDF, 360KB) that we presented in Barcelona last year.
Prof. Per Stenström, editor-in-chief, explains: In November 2005, the first in the series of International Conferences on High-Performance Embedded Architecture and Compilers was held in Barcelona. We were fortunate to attract close to a hundred submissions of which only 18 were selected for presentation. Among those, we asked the authors of the five most highly rated contributions to make extended versions of them. They all accepted to do that and their article appears in this issue.
Hedgehog on the walk
