Release Notes

This tool is in maintenance: new features will be frozened, existing bugs will be fixed. Functionality is integrated into the MDB Heterogeneous Migration Synchronization Suite

v3.7.1 (2024-03-21)

Bugfixs

Fix querying error when using function opengauss_version() in openGauss 2.0.0
Fix problem of data order when using time with microsecond as primary key in MySQL/MogDB comparion
Convert ctid to text to avoid driver error

v3.7.0 (2023-12-27)

Features

Add REMAP_OBJECT parameter to support comparing objects or tables with different names.
Add recognition of remapTable parameter in MTK (this parameter is deprecated and not recommended to use).

Improvements

In single MySQL scenarios, automatically set to MySQL time zone with customizable time zone name mapping.
Special handling of the +/- signs of time zones in openGauss/PG to account for their specificity.
Optimize the parsing of REMAP_SCHEMA parameter and object filtering.

Bugfixs

Fix a parameter identification problem when specifying MTK config file with some configuration items set to null.
Fix an error in parsing key when the source and target primary key columns do not match.
Fix MySQL CST server time zone not recognized by driver during connection
Fix comparison exception in versions of MySQL prior to 8.0.30 When invisible primary keys cannot queried in the data dictionary
Fix an error getting object list after DB2 specified excluded system table.
Fix a comparison issue in openGauss/MogDB caused by the case of the datacompatibility attribute.
Fix Oracle object query error when INCLUDE list is all %.

v3.6.5 (2023-11-30)

Improvements

Add field SOURCE_ERROR, TARGET_ERROR, CHECK_RESULT in result file with json format, CHECK_RESULT has value of SUCCEED or FAILED, which can be used to determine the final result

Bugfixs

Fix time formatting error in MogDB with version 5.0.0 and dolphin extension enabled

v3.6.4 (2023-11-22)

Features

Support to displaying the comparison elapsed time for each table in the result

Improvements

Add more information (inlcude real exception stacks) to the output when data compare error occured
Add collection types in openGauss when doing object comparison
Remove base table of materialized view when doing object comparison
Support to remove columns which is part of key columns, keep key columns but not compare them
Not to do partiton and sampling analysis when custom filter is specified
Split DB2 driver from executable program, avoid DB2 conencti error caused by the driver

Bugfixs

Fix time format error using "to_char" function in MogDB 5.0 with dbcompatibility B
Fix blanks in data when comparison of DB2 char/varchar and MogDB bytea
Fix PK mismatch when chr(0) in the value data

v3.6.3 (2023-11-01)

Features

Support saving result data when database query error or user interruptted occured (Saved to hidden file with prefix ".mvd_table_verify" and deleted after succeed)

Improvements

Expand column range for row-by-row KEY, support both primary key and unique key with not null columns
Adapted "schema.table:partition" config syntax in mtk, only read schema and table, ignore partition information
Expand dynamic sampling range, support dynamic sampling using both primary key or unique key columns
Remove default "--rtrim-varchar" setting in DB2, user can use it manually
Add msg file when packing executable programs to print more detailed error messages for DB2 database

Bugfixs

Add client encoding GB18030 when database in OG using GBK charset, avoid comparison failure due to unusual chinese characters
Fix the problem of SQL Server wrongly parsed index information .
Adjust the sorting column of DB2 with GBK charset, use hex sort, and add a second confirmation for the abnormal result of primary key table to avoid the inconsistency of data caused by the sorting rules.
Fix the problem that NULL values and empty strings have different sorting rules in different databases or modes when comparing data.
Fix the problem that the number of row locators is not displayed completely in the difference result of non-primary key table.
Fix an error when querying data in MySQL when calculating MD5 locally.

v3.6.2 (2023-10-11)

Features

Add "--local-md5" option to support calculating MD5 locally to avoid comparison failure caused by different database character sets.
Support comparing DB2 varchar/char type and MogDB bytea type data.

Improvements

Support exact column data difference location for some tables without primary key (which has unique index and unique index field is NOT NULL).
Remove the entry of column statistics comparison, and no longer deal with the problem of column statistics comparison.
Added uint4/uint8 type support introduced in openGauss 5.0.
"--upgrade" adapted to new naming convention.

Bugfixs

Fixed the problem that parsing results failed in mtk when the EXTRA field is empty in partitioned table comparison.
Fix an error in result summarization when comparing partitioned table.

v3.6.1 (2023-09-12)

Features

Add packages built on Kylin system

Improvements

Remove dependency on loguru, request modules.
Switch full table hash algorithm from md5 to xxhash128.
Add object type qualification when fetching Oracle object list (only fetch common and universal object types).

Bugfixs

Fixed the problem of not comparing data when comparing data only.
Fix connection timeout during MySQL large table comparison.
Fix memory overflow during MySQL large table comparison
Fix program error caused by object type inconsistency in some data-only comparison scenarios.
Fix an error in calculating overall hash in OG/MySQL when all columns are empty.

v3.6.0 (2023-07-07)

Features

Add option "-m/--mode", support summary signature compare with full table data, improve comparison efficiency， default is row-by-row compare

Improvements

The content of the data comparison results is expanded to adapt to the summary signature comparison results
Use function concat_ws instead of concat in MySQL, and remove coalesce function around all columns

Bugfixs

Fixed the issue that the column name in the constraint of MySQL was empty in some scenarios
Remove Oracle CLOB-related type support to avoid character concatenation errors and non-primary table sorting errors
Fix the program exception caused by partition boundary value type in MySQL
Fixed comparison failure in MySQL where year type cannot be formatted using date_format function

v3.5.2 (2023-06-09)

Improvements

Optimize MVD_TEMP directory deletion, consider customized MVD_TEMP variable, failure of temp deletion does not report error and affect the main program
Enable auto-commit for OG database connection by default
Automatically remove VARCHAR trailing spaces if the source database is DB2

Bugfixs

Repair the problem of max function failure caused by empty analysis time in dynamic sampling
Repair the error of finding unique key with column mapping (unhashable type: 'list')
Repair the comparison error caused by empty MySQL constraint column.
Fix the problem of inconsistent md5 value in Oracle database with non-UTF8 character set.

v3.5.1 (2023-04-26)

Features

Add --rtrim-varchar option to support removing trailing spaces from variable length strings before comparison

Improvements

Optimize the range supported by dynamic sampling, and support dynamic sampling according to the index leading column with the highest selectivity
Optimize time zone comparison logic, no time zone conversion by default, rely on database and driver's own time zone processing
Optimize the logic of Oracle Lib loading
Uniform fix the problem of pinpointing column differences when --row-feedback enabled, column difference query error does not affect the main process

Bugfixs

Repair the problem of missing where condition when comparing PG database
Optimize the temp file structure to avoid the problem of missing difference items when comparing table without primary key in same type of database in single process
Repair the comparison abnormality caused by case insensitivity in MySQL default collations
Repair the comparison failure caused by MySQL integer formatting with 0 at the end.
Repair the problem that the command line option --fraction-prec is not recognized
Repair the problem that the %s placeholder of bind variables in MySQL database conflicts with the placeholder of time formatting, which resulting in the error of querying back to the table
Repair the problem that the conversion of REMAP_SCHEMA objects to JSON is not serializable

v3.5.0 (2023-02-24)

Features

Support auto-generate SQL scripts for repairing target database based on source data (requires mtk in local environment, does not support SQL Server/MySQL tables without primary key)

Improvements

Change the format of difference result file to csv, which is easy to use and parsing
Optimize comparison logic to support mixed-case column names scenario

Bugfixs

Fix the problem that the software upgrades error when the version number has letters (compare_version function supports . /_split, support character comparison)
Fix the problem of incorrect comparison of multiple primary keys in openGauss environment.
Fix the problem of comparison failure for SQL Server tables without primary key
Fix the error of getting detailed abnormal columns in MySQL data comparison (Unread result found) by setting cursor to buffered=True.

v3.4.2 (2023-02-09)

Features

Support for precise comparison of column differences in difference rows via the --row-feedback parameter (precise display of specific column differences only for tables with the same primary key)

Improvements

Optimize the statistics logic of TOTAL_TABLES in the result summary data
Ignore triggers automatically generated by foreign keys in openGauss/PG databases (t.tgisinternal is false)
When parsing Oracle function-based indexes, the column wrapped in double quotes will be identified as normal, not function-base index column
Support parsing MTK configuration parameter "convertOracleIntegerToNumeric", which maps some NUMBER types in Oracle to bigint
Support parsing MTK configuration parameter "templateSeqName" to automatically recognize MySQL incremental columns as SEQUENCE of other databases
Support parsing MTK configuration parameter "skipColumnType"/"skipColumnName" to exclude custom columns, which can also be configured in the MVD configuration file
Support parsing MTK configuration parameter "excludeSysTable", exclude custom system table, which can also be configured in the MVD configuration file
Support parsing MTK configuration parameter "autoAddMySQLAutoIncr" to ignore the automatically added PK_ID and its related information.
Support resolving MTK configuration parameter "disableFKCons" to ignore foreign key comparison.
Support parsing MTK configuration parameter "dataOnly"/"schemaOnly"

Bugfixs

Repair the comparison error in SQL Server caused by the loss of table name processing
Repair the error reported in Informix due to unprocessed partition table function.
Repair the comparison failure caused by the return string of table size query in Informix.
Repair the comparison problem caused by the function/procedure in package in openGauss is also stored in pg_proc (propackageid=0)
Repair the problem that PACKAGE BODY comparison fails when comparing Oracle vs openGauss.
Repair the design that the trigger and trigger function are separated from each other in openGauss/PG, and ignore the trigger function.
Repair the problem that MogDB custom data types with schema information when converting ::regtype cause comparison failure.
Repair the exception when there is no TABLE in the object type.
Repair the problem that data comparison cannot be performed when only data is compared.

v3.4.1 (2023-02-02)

Features

In the JSON result, add DIFF_FILE data to each table, recording the path where the table's diff data is located

Improvements

Add option --ora-float-prec to specify float precision in Oracle separately
Optimize the recognition rules for COLUMN_LIST and DATA_FILTER in configuration files
Support to wrap schema and object name with quotes in -T single table comparison mode to preserve their case
Add recognition and comparison of mixed case objects in each database when comparing in batch mode
Update the comparison precision of float/double field data, the default is adaptive (use max precision -1)
Update sample configuration file

Bugfixs

Fix the problem that when all columns of table without primary key in openGauss are NULL, the MD5 value is also NULL, resulting in order problem, thus causing the failure of comparing this part of data
Repair the potential problem that some newly added functions in MySQL do not add strip() to schema and table name.
Repair the problem that the whole comparison task fails because of Oracle cannot decode the character, and use hexadecimal system to identify the abnormal characters.
Repair the problem of incorrect handling of MAXVALUE boundary in partition slice comparison under some scenarios.
Repair the data inconsistency caused by the character encoding problem when calculating the MD5 value internally for NCHAR/NVARCHAR data type in Oracle database.
Repair the problem that the TIMESTAMP partition field is not parsed correctly in Oracle partitioning.
Repair the statistical problem of DATA_DIFF_TABLES in the comparison result, and the data difference should be the both of two tables in the source and target database.
Repair the abnormality of Type/TypeBody object when comparing with Oracle and MogDB.
Repair the problem of MogDB number overflow and adjust Oracle number to string using TM9 format.
Repair the problem of missing field expressions introduced due to the processing of Oracle time zone.
Repair the problem of Oracle Lib searching and loading in macOS (the problem of mixing multiple /tmp/_MEI directories)
Repair the data abnormality caused by inconsistent time zones in the comparison of Oracle timestamps with time zones.
Repair the problem that MVD_[SOURCE/TARGET]_DSN environment variable cannot be read in the official release version of MVD 3.4.0.

v3.4.0 (2023-01-11)

Features

Support dynamic sampling interval division based on real data distribution, and dynamic partial data comparison by percentage
Supports parallel comparison of partitioned tables, currently only supports RANGE partitions with a single number or time column
Partial comparison of a single table is supported by the --data-filter option, and partial data comparison of a single condition can coexist with partitioned parallelism and dynamic sampling parallelism
Support for specifying multiple filters via the --data-filter option to enable manual concurrent comparison of single tables
The data comparison result shows the comparison mode used in the current table, including: Split/Sample/Partition/Partial/Normal, etc.
Add the --detail-mode option to support displaying table results without data differences in the results
Support reading database connection information from environment variables (MVD_SOURCE_DSN/MVD_TARGET_DSN), support database password decryption
Support automatic slicing of full data based on primary key (20 slices), by default consider slicing only when the table size is bigger than 10GB

Improvements

Support for parsing custom data slicing rules in MTK configuration files (object.tableSplit)
Supports different column restrictions and mappings for different tables via option "--column-list"
Adjust the option psrsing logic to support the execution of both command line options and configuration files
In openGauss, the ctid is not unique in partitioned table, when table has no primary key, the difference file concat where condition with ctid
In Oracle, support calculating and sorting MD5 data within database when row length less than 4000
Add summary data to the final result

Bugfixs

Fix error when table does not exists in single table mode (with -T option used)
Fix memory problem in function openGauss.query()

v3.3.1 (2022-11-25)

Features

Support customize precision of float,double and fraction, max fraction in informix adjust to 5

Improvements

Optimizing MySQL Connections: try to connect in two method, default auth-plugin and 'mysql_native_password' auth-plugin
Optimizing the logical of packing scripts, reduce the amount of external lib files and the size of executable program
Consider float/double precision in column statistics comparasion

Bugfixs

Fix problem: missing one column in md5 hash calculation when compare primary-key table in Informix
Fix problem: data type of money direct convert to string with dollar sign
Fix problem: cannot recognize varchar and tinyint data type in MySQL data comparasion
Fix problem: concat null returns null in openGauss with dbcompatibility=B
Fix problem: coalesce(null::int, '') returns 0 in openGauss with dbcompatibility=B

v3.3.0 (2022-11-21)

Features

Support Informix: basic object comparasion and md5 comparasion for table row data

Improvements

Support boolean column type in data comparasion for openGauss

Bugfixs

Fix row value assignment error in data comparasion for openGauss
Repair the problem that openGauss data comparison does not recognize the text field

v3.2.0 (2022-11-03)

Features

Support SQL SERVER: basic object comparasion and md5 comparasion for table row data
Support receive process info and result by callback interface

Improvements

Tuning Oracle library initial logical in macOS
Consider customer column name mapping in data structure comparasion
Support large sequence objects in openGauss

Bugfixs

Fix error in data handling when column data type is money
Fix error in statistics comparasion when column-list options does not set
Fix MySQL driver error: ImportError: No localization support for language 'eng'

v3.1.1 (2022-10-12)

Bugfixs

Fix the problem caused by splicing empty strings in MogDB/openGauss/MySQL, compatible with mariadb

v3.1.0 (2022-10-10)

Improvements

License version upgrade to v31, older license file does not supported

Bugfixs

Fix the problem caused by splicing empty strings in MogDB/openGauss when comparing primary key tables

v3.0.1 (2022-09-23)

Features

Add "--upgrade" option to support command self-upgrade
Support comparison for PostgreSQL
Support remap for schema and table name in single table mode (with "-T" option used)
Support remap and filter for columns in single table mode (with "-T" option used)

Improvements

Query optimization for Oracle table without primary key
Data query sorting rules adjustment (with primary key sorted by primary key, without primary key sorted by MD5 value)

Bugfixs

Repair the query error caused by multi-byte characters in Oracle fields under some scenarios

v3.0.0 (2022-06-09)

Features

Add options: "-L", "--license", "--apply-licnese", used for apply and verify license
Add option: "--column-list", support partial column comparion

Improvements

Use "--debug-md5" option can output primary key in source and target database
When comparing with statistics method, filter columns before query
Adjust the display of empty characters in object names (use double quotes to wrap them)
Support for MySQL with object names without space compared to other databases with space in name
Add debug information for openGauss SQL executions

Bugfixs

Fix "NULL" (None in python) value returns when comparision with primary key
Remove "public" schema out of system schema list in openGauss

v2.4.8 (2022-04-22)

Feature

When comparing objects, MogDB adds Package objects (2.1 and later versions) and SYNONYM objects comparison
Add recognition of OS local variables (TMPDIR, TMP) for storing temporary data during program runtime when comparing Oracle/DB2 databases
Add -Z option to manually specify the time zone used for data comparison in different databases at runtime

Function Optimization

Fine-tuned final results presentation for better readability
Optimize the output of the --debug-md5 option during data comparison to provide a more detailed view of the bilateral data comparison process
Optimize -T mode to support -r specified Schema mapping even in single table comparison mode

Bugfix

Fix the issue of abnormal comparison results caused by different sorting rules of primary key tables in openGauss/MogDB when querying
Fix the issue of program error caused by the failure of primary key identification in the comparison of non-primary key table.
Fix the issue of inconsistent sorting data due to mixed case of primary key field names in different databases ("TID").
Fix the issue of inconsistent data in the unilateral database which cannot be correctly identified.
Fix the issue that the comparison result of primary key table is incorrect due to the space at the end of the primary key field.
Fix the issue that the query field appears several times in Oracle/DB2 (primary key table), which leads to the query error.
Fix the issue of incorrect data comparison caused by trailing space of char type in MySQL database.

v2.4.0 (2021-12-20)

Feature

Add the category option (-C/--catagory): supports object structure comparison only and data comparison only.
Add the debug option (--debug-md5): outputs the original value of the Python MD5 computing result during data comparison.
Add the schema remapping option (--remap-schema): supports data comparison even if the name of a schema in a source or target database is changed.

Function Optimization

Optimize the data comparison logic: For tables with primary keys, comparison is implemented by primary key. Additionally, the values of the primary keys and the MD5 computing results of the source and target databases will be output in the result.
Optimize the output result: The display of the object structure comparison result and data comparison result is rewritten, thereby facilitating reading and understanding.
Optimize the DB2 connection mode: During DB2 database connection, more authentication methods can be used (SERVER_ENCRYPT, SERVER, CLIENT, and default).

Bugfix

Fix the data comparison failure caused by invisible characters chr(0), and add the -z/--zero-char option for specifying the conversion character of chr(0) during migration.
Fix the issue that the numeral modulo operator is unavailable in some versions during restoration of the DB2 database.

v2.3.2 (2021-11-08)

Bugfix

Resolve the table constraint comparison error in MySQL earlier than 8.0. Therefore, MySQL 5.5, 5.7, and 8.0 are supported.
Resolve the problem that the data types of return values in the Python driver for the fields in MySQL of different versions are different.

v2.3.1

Bugfix

Resolve the error that the program is abnormal because the return value of the driver is str/byte for the MySQL query field type in some versions.
Resolve the error that the return value of the lsb_release -a command is incorrect when the MySQL driver is to obtain the OS version in SUSUE.

v2.3.0

Feature

Support the sha256 authentication method for openGauss/MogDB in Linux.