Running and Results

This page describes how the benchmark tests have been run and how results have been calculated.

The results themselves are presented on the results pages.

Hardware and Software

The benchmark has been run on the following hardware:

  • CPU - Intel Core i3-2120 @ 3.30GHz
  • RAM - 12GB
  • Hard Disk - Seagate Barracuda 7200.11 SATA 2 1.5TB
and using the following software:
  • Microsoft Windows 7 Pro
  • Sun Java JDK 6 update 29

Running the Test

Several actions have been taken in order to improve the accuracy of the benchmark results:

  • Every test has been run in a new separate process with a new JVM. This is essential to prevent a situation in which one test run affects the other. For example one test run uses the Java heap memory, but garbage collecting for that activity happens only on the run of the next test run.
  • The time of every test run has been divided into 2 phases. The first phase was always ignored. Only the performance in the second phase has been taken into account. This way, initialization activities are excluded from time measurement, including class loading, class enhancement by a Java Agent, JIT compilation, Java HotSpot JVM optimization (e.g. inlining), and database schema generation by ORM tools.
  • All non essential background processes have been disabled, including Anti Virus, Firewall and File Indexing.
  • To minimize the effect of essential background operating system processes that cannot be disabled, every test has been run twice, and only the better result has been taken into account. The other result was simply ignored, since it might be affected by a background operating system process, and the better result more likely reflects better the real capability of the tested products in the test.

Result Calculation

To avoid confusion, all the results in this benchmark keep the rule of "higher is better".

Absolute results represent:

  • In the context of speed performance - operations per second.
  • In the context of space efficiency - entity objects per MB.

For every absolute result a normalized result (between 0 to 100) is calculated as: 

    100 * {absolute result} / {best absolute result}
where {best absolute result} is the highest absolute result in that combination of test, operation and batch mode (i.e. the absolute result of the best DBMS/JPA in that test). The normalized results are used in charts and as a base for average calculations.

Finally, when presenting results of several different test runs. For example, the main result page that shows average results of all the performance tests, average scores are calculated as arithmetic means of the normalized results in all the included test runs.