Dear Temporal team,
I went through a lot of threads related to cross DCs setup but still need some advices…
Let me start with 2 requirements/constraints:
- data loss is not affordable so data consistency is the priority
- RTO has to be close to 0 but still, less important than consistency
- we have 2 datacenters (not in public cloud)
- the latency between our 2 DCs is less than 10ms (~6ms avg)
- Postgres stretch cluster with synchronous replication is available (we use a 3rd DC for etcd for the quorum to avoid split brain)
My understanding is that:
- Because of the strong consistency requirement, multi-cluster replication is not an option
- It is NOT possible to have 2 separate Temporal clusters (up and active/active or active/passive) with the same database
So, we would like to setup a Temporal stretched cluster…
In this scenario, we have a single Temporal Cluster spanning across our 2 DCs.
The whole cluster relies on our Postgres stretched cluster for persistence (it would also work with Cassandra is need be).
- Is this a viable option?
- With only 2 DCs for Temporal, is it split brain resistant. In this case, I am assuming that the DB is still up and running but the communication between the Temporal nodes in both DCs is broken but the nodes on both sides can read/write into the DB.
Would you have an idea of what would happen…
- If not split brain resistant, is there a way to make it split brain resistant with 2 DCs or any thought on how to mitigate the risk?
It seems that this is what’s discussed in this thread (not 100% sure): temporal-with-mysql-how-to-survive-master-slave-failovers
Many thanks in advance for your support.
Best regards, Seb.