Multi-Cluster Replication config changes corrupting database

With the 1.14 release and future 1.15 release, we are moving cluster metadata loading from yaml static config to API dynamic config. To make this process airstream, we initialize new cluster metadata from yaml config once during startup.

If you already start the service with a cluster metadata (secondary or standby) and later update some critical fields in the cluster metadata. It gives you the errors you mentioned above.

There are two options for you as you are experiment the multi-cluster setup.

Option 1: Update cluster metadata from admin tctl

./tctl adm cl upsert-remote-cluster --frontend_address <rpcAddress> --enable_connection <true/false>

This will make a call to the rpcAddress endpoint to fetch the target cluster metadata. You need to make sure the enableGlobalNamespace is true, the initialFailoverVersion is not change, and the failoverVersionIncrement values are equal. (If not, you have to do delete it first, see option 2)

Option 2: Delete the invalid cluster metadata first, use admin tctl or yaml to set the correct cluster metadata.

./tctl adm cl  remove-remote-cluster --cluster <cluster name>

Note: make sure to do this operation on both clusters.