Multicluster replication with three clusters

We’re trying to setup multi-cluster replication between three self-hosted Temporal clusters and get errors:

Operation failed with internal error. UpdateAckLevel operation encountered concurrent write.

I tracked where it comes from, and it looks like it is emitted from this line. It updates queue_metadata row when returning replication messages via admin API. As there are two clusters asking for replication messages, I assume it tries to increment row field version that is already incremented while serving request from another cluster (query), and hence we get rowsAffected != 1 , i. e. it is 0. Does it mean that replication to two or more clusters does not work? We use server version v1.22.7.

What type of database are you using?

We’re using Vitess (MySQL).

I see. Let me double check the code. From my current understanding, this is a transient error and retry should do the work. We do support 2+ clusters but this error can show up more often with multi cluster setup.

We deployed three cluster replication anyway and all seems to be working fine, except that this error appears every couple of minutes on random clusters.

This error should be benign. We already filter out this ConditionFailedError. It should not be logged out.