Test Description

This benchmark is composed of 6 tests. Each test checks the 4 CRUD operations (Create, Retrieve, Update and Delete) and storage efficiency (by measuring the database size). Every operation is run in two batch size modes (large and small retrieval and transaction size). In total, when counting all the combinations of tests/operations/modes this benchmark includes 58 performance tests and 6 storage efficiency checks.

In the Result Page you can see the results of every one of these tests, as well as averages of all or a subset of these tests.

Benchmark Tests

This benchmark is composed of the following tests:

  • Basic Person Test
    The basic test uses JPA with one entity class that represents a person. The Person class contains an auto generated Integer primary key field, 10 String persistent fields, 3 date persistent fields and one int persistent field, representing the person details. This simple object model of one entity class does not include inheritance, collections, relationship or indexes.
  • Element Collection Test
    The Element Collection Test is based on the Basic Person test with the following change. Instead of a simple String field for the phone number, a List<String> collection field containing 1-3 phone numbers is used. This tiny change in the object model has a larger effect on the ORM/DBMS side, because of the flat nature of records in ordinary RDBMS tables.
  • Inheritance Test
    The Inheritance test is based on the Basic Person test with the following change. Instead of one Person entity class, an hierarchy of 3 classes is used. The original Person fields are divided between the 3 classes in the inheritance hierarchy.
  • Indexing Test
    The Indexing test is based on the Basic Person test with one addition. An index is defined for one of the the String fields (the person's last name). Because JPA does not define a standard annotation for defining an index, vendor annotations are used. EclipseLink was excluded from this test because it does not provide an annotation for defining an index.
  • Graph (Binary Tree) Test
    The Graph test uses JPA with a different object model that represents a binary tree. The Node entity class, which represents a node in the tree, includes an Integer primary key field, two persistent reference fields to child nodes and a simple int persistent field with data. Each tree contains 100 nodes.
  • Multithreading Test
    The Multithreading test is identical to the Basic Person test except that all the operations are run using 5 concurrent threads (rather than in a single thread as the other tests).

Test Operations

Each test executes the following 4 CRUD operations:

  • Persisting Entity Objects
    Every test starts by populating the database. 100,000 entity objects with randomized data are persisted to the database. This is the most time consuming operation in the benchmark because persisting objects to the database using some of the tested DBMS/ORM combinations is very slow (otherwise we could have used larger databases).
  • Retrieving Entity Objects
    After populating the database, retrieval of entity objects from the database is tested. Objects are retrieved in this step from the database by using the primary key (random primary key values are selected for retrieval).
  • Updating Entity Objects
    The next tested CRUD operation is update. Random entity objects are retrieved from the database (by their primary key). The objects are updated in memory and then the updates are committed to the database.
  • Removing Entity Objects
    The last tested CRUD operation is remove. Random entity objects are retrieved from the database (by their primary key) and then deleted from the database.

JPQL Queries
Retrieval by primary key (which is described above) uses JPQL query (because find is limited to a single entity object).
The Person based tests (all but the Graph/Binary Tree test), includes an additional JPQL query on a simple String field (the person's last name). The tested query retrieves all the persons whose last name starts with a specified random prefix.
Retrieval by string prefix is very common (for example, in AJAX based web applications with auto complete forms).

Storage Efficiency
The disk space that a database consumes is measured when the database population is completed. Only results of this check for embedded databases are published, since disk space measurement of databases in server mode by this benchmark is inaccurate.

Batch Modes (Small / Large)

All the tests have been run in the following two batch mode:

  • Small Transaction Size / Retrieval Size
    In this mode transactions are committed after every 5 insert/update/delete operations and entity objects are retrieved in groups of 5 entity objects. Except in the Graph/Binary Tree test, in which every single tree has 100 nodes, so 100 is used instead of 5.
  • Large Transaction Size / Retrieval Size
    In this mode transactions are committed after every 5,000 insert/update/delete operations and entity objects are retrieved in groups of 5,000 entity objects.