MOT Sample TPC-C Benchmark

TPC-C Introduction

The TPC-C Benchmark is an industry standard benchmark for measuring the performance of Online Transaction Processing (OLTP) systems. It is based on a complex database and a number of different transaction types that are executed on it. TPC-C is both a hardware-independent and a software-independent benchmark and can thus be run on every test platform. An official overview of the benchmark model can be found at the tpc.org website here - http://www.tpc.org/default5.asp.

The database consists of nine tables of various structures and thus also nine types of data records. The size and quantity of the data records varies per table. A mix of five concurrent transactions of varying types and complexities is executed on the database, which are largely online or in part queued for deferred batch processing. Because these tables compete for limited system resources, many system components are stressed and data changes are executed in a variety of ways.

Table 1 TPC-C Database Structure

Table	Number of Entries
Warehouse	n
Item	100,000
Stock	n x 100,000
District	n x 10
Customer	3,000 per district, 30,000 per warehouse
Order	Number of customers (initial value)
New order	30% of the orders (initial value)
Order line	~ 10 per order
History	Number of customers (initial value)

The transaction mix represents the complete business processing of an order - from its entry through to its delivery. More specifically, the provided mix is designed to produce an equal number of new-order transactions and payment transactions and to produce a single delivery transaction, a single order-status transaction and a single stock-level transaction for every ten new-order transactions.

Table 2 TPC-C Transactions Ratio

Transaction Level ≥ 4%	Share of All Transactions
TPC-C New order	≤ 45%
Payment	≥ 43%
Order status	≥ 4%
Delivery	≥ 4% (batch)
Stock level	≥ 4%

There are two ways to execute the transactions - as stored procedures (which allow higher throughput) and in standard interactive SQL mode.

Performance Metric - tpm-C

The tpm-C metric is the number of new-order transactions executed per minute. Given the required mix and a wide range of complexity and types among the transactions, this metric most closely simulates a comprehensive business activity, not just one or two transactions or computer operations. For this reason, the tpm-C metric is considered to be a measure of business throughput.

The tpm-C unit of measure is expressed as transactions-per-minute-C, whereas "C" stands for TPC-C specific benchmark.

NOTE: The official TPC-C Benchmark specification can be accessed at - http://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_v5.11.0.pdf. Some of the rules of this specification are generally not fulfilled in the industry, because they are too strict for industry reality. For example, Scaling rules - (a) tpm-C / Warehouse must be >9 and <12.86 (implying that a very high warehouses rate is required in order to achieve a high tpm-C rate, which also means that an extremely large database and memory capacity are required); and (b) 10x terminals x Warehouses (implying a huge quantity of simulated clients).

System-Level Optimization

Follow the instructions in the MOT Server Optimization - x86 section. The following section describes the key system-level optimizations for deploying the MogDB database on a Huawei Taishan server and on a Euler 2.8 operating system for ultimate performance.

BenchmarkSQL - An Open-Source TPC-C Tool

For example, to test TPCC, the BenchmarkSQL can be used, as follows -

• Download benchmarksql from the following link - https://osdn.net/frs/g_redir.php?m=kent&f=benchmarksql%2Fbenchmarksql-5.0.zip
• The schema creation scripts in the benchmarksql tool need to be adjusted to MOT syntax and unsupported DDLs need to be avoided. The adjusted scripts can be directly downloaded from the following link - https://opengauss.obs.cn-south-1.myhuaweicloud.com/1.0.0/MOT-TPCC-Benchmark.tar.gz. The contents of this tar file includes sql.common.mogdb.mot folder and jTPCCTData.java file as well as a sample configuration file postgresql.conf and a TPCC properties file props.mot for reference.
• Place the sql.common.mogdb.mot folder in the same level as sql.common under run folder and replace the file src/client/jTPCCTData.java with the downloaded java file.
• Edit the file runDatabaseBuild.sh under run folder to remove extraHistID from AFTER_LOAD list to avoid unsupported alter table DDL.
• Replace the JDBC driver under lib/postgres folder with the MogDB JDBC driver available from the following link - https://opengauss.org/en/download/.

The only change done in the downloaded java file (compared to the original one) was to comment the error log printing for serialization and duplicate key errors. These errors are normal in case of MOT, since it uses Optimistic Concurrency Control (OCC) mechanism.

NOTE: The benchmark test is executed using a standard interactive SQL mode without stored procedures.

Running the Benchmark

Anyone can run the benchmark by starting up the server and running the benchmarksql scripts.

To run the benchmark -

1. Go to the benchmarksql run folder and rename sql.common to sql.common.orig.
2. Create a link sql.common to sql.common.mogdb.mot in order to test MOT.
3. Start up the database server.
4. Configure the props.pg file in the client.
5. Run the benchmark.

Results Report

• Results in CLI

  BenchmarkSQL results should appear as follows -

  

  Over time, the benchmark measures and averages the committed transactions. The example above benchmarks for two minutes.

  The score is 2.71M tmp-C (new-orders per-minute), which is 45% of the total committed transactions, meaning the tpmTOTAL.

• Detailed Result Report

  The following is an example of a detailed result report -

  Figure 1 Detailed Result Report

  

  

  BenchmarkSQL collects detailed performance statistics and operating system performance data (if configured).

  This information can show the latency of the queries, and thus expose bottlenecks related to storage/network/CPU.

• Results of TPC-C of MOT on Huawei Taishan 2480

  Our TPC-C benchmark dated 01-May-2020 with an MogDB database installed on Taishan 2480 server (a 4-socket ARM/Kunpeng server), achieved a throughput of 4.79M tpm-C.

  A near linear scalability was demonstrated, as shown below -

  Figure 2 Results of TPC-C of MOT on Huawei Taishan 2480