Support bidirectional data replication between two TiDB clusters

[shenli]

Gravity could replicate data between two MySQL clusters bidirectionally and could read TiDB's binlog. So I guess it is not hard to support bidirectional data replication between two TiDB Clusters.

[Ryan-Git]

@shenli
Yes. Could you provide a docker image that starts TiDB cluster and kafka with proper configuration(pumper, drainer, etc)? It would be appreciated if binlog could be subscribed from it directly.

[WangXiangUSTC]

can use docker-compose to start a TiDB cluster with binlog, can see https://github.com/pingcap/tidb-docker-compose, use docker-compose-binlog.yml this file, and need update drainer's config file to send binlog to kafka.

[IANTHEREAL]

@Ryan-Git are anything going well?

[Ryan-Git]

@GregoryIan I've roughly finished the change. Need more time for testing.

btw, there's no unique key info in the messages. It will cause some problems when writing to target in parallel.

[WangXiangUSTC]

@Ryan-Git do you means find a bug when table without unique key in gravity?

[Ryan-Git]

@WangXiangUSTC

For mysql, we consider new and old values for both primary keys and unique keys as "data dependency". A message has duplicate dependency with ongoing ones will be delayed until they finished. It's just like lock/latch mechanism in databases.

for example, there's one row with pk = 1 and uk = 1. Two transactions happen in following sequence

delete from t where uk = 1;
insert into t(pk, uk) values (2, 1);

They should be executed in sequence. But if we only know primary key, it's different(1 and 2). So gravity is allowed to output them in arbitrary order.

[july2993]

@WangXiangUSTC

For mysql, we consider new and old values for both primary keys and unique keys as "data dependency". A message has duplicate dependency with ongoing ones will be delayed until they finished. It's just like lock/latch mechanism in databases.

for example, there's one row with pk = 1 and uk = 1. Two transactions happen in following sequence

delete from t where uk = 1;
insert into t(pk, uk) values (2, 1);

They should be executed in sequence. But if we only know primary key, it's different(1 and 2). So gravity is allowed to output them in arbitrary order.

can we get the schema from downstream?
another choice we can add the unique key info in the messages to address this problem.

[Ryan-Git]

Getting the schema from downstream might work, but

• Logically, data dependency is decided by source data store, not the target. Though it seems identical in this specific scenario, it's still error prone.

• In practice, the internal message structure is built by input component. When it reaches output, it has been scheduled thus too late. Connecting to the output data store in input component is ridiculous.

One way to get around this is connecting to the source data store. But the message and schema may not match unless we keep a schema history. That's another story though(#174 ). I think add the unique key info in the messages is a better solution for TiDB.

Temporally, we can set number of workers to one, handle the messages sequentially, just like drainer outputs to only one partition. But the throughput would be a problem.

btw, do we need to support DDL?

[july2993]

Getting the schema from downstream might work, but

• Logically, data dependency is decided by source data store, not the target. Though it seems identical in this specific scenario, it's still error prone.
• In practice, the internal message structure is built by input component. When it reaches output, it has been scheduled thus too late. Connecting to the output data store in input component is ridiculous.

One way to get around this is connecting to the source data store. But the message and schema may not match unless we keep a schema history. That's another story though(#174 ). I think add the unique key info in the messages is a better solution for TiDB.

Temporally, we can set number of workers to one, handle the messages sequentially, just like drainer outputs to only one partition. But the throughput would be a problem.

btw, do we need to support DDL?

I think i can add the unique key info in the messages later.

Is the DDL supported for other sources like mysql ?

[Ryan-Git]

I think i can add the unique key info in the messages later.

Is the DDL supported for other sources like mysql ?

Yes, partially supported.

[Ryan-Git]

@july2993
One more question. How could we describe composite key? The ColumnInfo type supports only single column primary key I think.

[july2993]

@july2993
One more question. How could we describe composite key? The ColumnInfo type supports only single column primary key I think.

all column belong to PK will be mark as IsPrimaryKey for composite key,
I will add the unique key info in pingcap/tidb-tools#310, after that, i think you can ignore the IsPrimaryKey flag, could you take a look?