Performance Test Overview

Test Indicators

Data Volume

Each repository contains 70 MB of data. To simulate the actual data environment, 1000 repositories are created during initiation. After related indexes are added, the total size of data reaches 100 GB or so.

Number of Concurrent Requests

This is to simulate the real user behavior and make the number of online user requests from the clients reach a certain number.

Average System Workload

System workload indicates how busy the system CPU is, for example, how many processes are there waiting for being scheduled by the CPU, which reflects the pressure of the whole system. However, average system workload indicates the average number of processes in run queues in a specified time period, generally 1, 5, or 10 minutes.

CPU Usage

CPU usage indicates the quantity of CPU resources occupied by the running programs, that is, the quantity of running programs on a server at a time point. High CPU usage indicates that a server runs a plenty of programs and vice versa. Whether the CPU usage is high or low is directly related to the CPU performance. CPU usage can determine whether a CPU reaches its bottleneck.

IOPS

IOPS (Input/Output Operations Per Second) is a method for measuring the performance of a computer storage device, such as HDDs, SSDs, or SANs. It refers to the number of write/read times per second, which is one of the major indicators for measuring the disk performance. IOPS indicates the number of I/O requests that can be processed by the system in unit time. The I/O requests usually refer to data write/read requests.

IO Latency

IO latency is also called IO response time, referring the time period from the time when the OS kernel sends a write/read command to the time when the OS kernel receives the IO response. Time for operating single IO indicates only the time of processing single IO in a disk. However, IO latency includes even the time of waiting for being processed for an IO in a queue.

tpmC

TPC-C uses three performance and price metrics. TPC-C is used to measure performance with the unit of tpmC (Transactions Per Minute). C refers to the C benchmark program of TPC. tpmC indicates the number of new orders that are processed by the system per minute.

Test Program Preparation

Test Tools

BenchmarkSQL is a typical open source database test tool with a built-in TPC-C test script which can be directly used for testing PostgreSQL, MySQL, Oracle, and SQL Server databases. It is used for testing TPC-C of OLTP (Online Transaction Processing) via JDBC (Java Database Connectivity).

Test Specifications

TPC-C offers OLTP system test specifications and tests an OLTP system using a merchandising model. Transactions are divided into five categories, the content and characteristics of which are described as follows:

• NewOrder - Generation of new orders

  Transaction content: For any one client, select 5 to 15 commodities from a specified repository and create new orders with 1% of the orders that fails the operation and requires rollback.

  Main characteristics: middleweight, frequent write/read operations, and high response speed

• Payment - Payment of order

  Transaction content: For any one client, select a region and customers there from a specified repository randomly, pay an order with a random amount of money, and record this operation.

  Main characteristics: lightweight, frequent write/read operations, and high response speed。

• OrderStatus - Query of latest orders

  Transaction content: For any one client, select a region and customers there from a specified repository randomly, read the last order, and display the status of each commodity in the order.

  Main characteristics: middleweight, low read-only frequency, and high response speed

• Delivery - Delivery of packages

  Transaction content: For any one client, select a delivery package randomly, update the account balance of the user whose order is being processed, and delete the order from the new order list.

  Main characteristics: 1 to 10 concurrent orders in a batch, low write/read frequency, and loose response time

• StockLevel - Analysis of stock status

  Transaction content: For any one client, select the last 20 orders from a specified repository of a region, check the stock of all commodities of the order list, and calculate and display the stock of all commodities with its stock level lower than the threshold generated randomly.

  Main characteristics: heavyweight, low read-only frequency, and loose response time

Test Rules

Before the test, TPC-C Benchmark specifies the initial status of a database, that is, the rule of data being generated in the database. The ITEM table includes a fixed number (100,000) of commodities. The number of warehouses can be adjusted. The initialized WAREHOUSE table includes 1000 records in this test.

• The STOCK table contains 1000 x 100,000 records. (Each warehouse contains 100,000 commodities.)
• The DISTRICT table contains 1000 x 10 records. (Each warehouse provides services for 10 regions.)
• The CUSTOMER table contains 1000 x 10 x 3000 records. (There are 3000 customers in each region.)
• The HISTORY table contains 1000 x 10 x 3000 records. (Each customer has one transaction history.)
• The ORDER table contains 1000 x 10 x 3000 records (there are 3000 orders in each region). The last 900 orders generated are added to the NEW-ORDER table. Each order generates 5 to 15 ORDER-LINE records.

TPC-C uses tpmC to measure the maximum qualified throughput. The number of transactions depends on that of the new order transactions, that is, the number of new orders processed per minute.