Multicluster replication with three clusters

Community Support Server Deployment
1

We’re trying to setup multi-cluster replication between three self-hosted Temporal clusters and get errors:

Operation failed with internal error. UpdateAckLevel operation encountered concurrent write.

I tracked where it comes from, and it looks like it is emitted from this line. It updates queue_metadata row when returning replication messages via admin API. As there are two clusters asking for replication messages, I assume it tries to increment row field version that is already incremented while serving request from another cluster (query), and hence we get rowsAffected != 1 , i. e. it is 0. Does it mean that replication to two or more clusters does not work? We use server version v1.22.7.

2

What type of database are you using?

3

We’re using Vitess (MySQL).

4

I see. Let me double check the code. From my current understanding, this is a transient error and retry should do the work. We do support 2+ clusters but this error can show up more often with multi cluster setup.

5

We deployed three cluster replication anyway and all seems to be working fine, except that this error appears every couple of minutes on random clusters.

6

This error should be benign. We already filter out this ConditionFailedError. It should not be logged out.