Data Migration Architecture

This document introduces the architecture of Data Migration (DM).

DM consists of three components: DM-master, DM-worker, and dmctl.

Data Migration architecture

Architecture components

DM-master

DM-master manages and schedules the operations of data migration tasks.

Storing the topology information of the DM cluster
Monitoring the running state of DM-worker processes
Monitoring the running state of data migration tasks
Providing a unified portal for the management of data migration tasks
Coordinating the DDL migration of sharded tables in each instance under the sharding scenario

DM-worker

DM-worker executes specific data migration tasks.

Persisting the binlog data to the local storage
Storing the configuration information of the data migration subtasks
Orchestrating the operation of the data migration subtasks
Monitoring the running state of the data migration subtasks

For more details of DM-worker, see DM-worker Introduction.

dmctl

dmctl is a command line tool used to control the DM cluster.

Creating, updating, or dropping data migration tasks
Checking the state of data migration tasks
Handling errors of data migration tasks
Verifying the configuration correctness of data migration tasks

Architecture features

High availability

When you deploy multiple DM-master nodes, all DM-master nodes use the embedded etcd to form a cluster. The DM-master cluster is used to store metadata such as cluster node information and task configuration. The leader node elected through etcd is used to provide services such as cluster management and data migration task management. Therefore, if the number of available DM-master nodes exceeds half of the deployed nodes, the DM cluster can normally provide services.

When the number of deployed DM-worker nodes exceeds the number of upstream MySQL/MariaDB nodes, the extra DM-worker nodes are idle by default. If a DM-worker node goes offline or is isolated from the DM-master leader, DM-master automatically schedules data migration tasks of the original DM-worker node to other idle DM-worker nodes. (If a DM-worker node is isolated, it automatically stops the data migration tasks on it); if there are no available idle DM-worker nodes, the data migration tasks of the original DM-worker are temporarily hung until one DM-worker node becomes idle, and then the tasks are automatically resumed.

Note

When the data migration task is in the process of full export or import, the migration task does not support high availability. Here are the main reasons:

For the full export, MySQL does not support exporting from a specific snapshot point yet. This means that after the data migration task is rescheduled or restarted, the export cannot resume from the previous interruption point.
For the full import, DM-worker does not support reading exported full data across the nodes yet. This means that after the data migration task is scheduled to a new DM-worker node, you cannot read the exported full data on the original DM-worker node before the scheduling happens.