Hi all, question about replication topology at scale.
By design, Temporal clusters can be connected in a full mesh, where every cluster can replicate with every other cluster. But enabling replication streaming between two clusters opens a sender and receiver connection per shard, regardless of whether any namespace is actually replicating between them. With 4096 shards and 4+ clusters in a mesh, that’s a lot of idle connections and per-shard bookkeeping accumulating fast.
The appeal of a full mesh is operational flexibility: it’s easy to load-balance and move namespaces between any pair of clusters without re-plumbing connectivity. The downside we’re worried about is twofold:
- Resource overhead — idle sender/receiver pairs for cluster pairs with no active namespace replication between them.
- Blast radius — whether a slow/degraded replication stream on one cluster pair can starve or throttle replication for unrelated namespaces on a different pair, due to shared per-cluster resources (connection pools, goroutines, etc.).
The alternative is connecting clusters only in the pairs that actually need replication rather than a full mesh. Less flexible for future namespace moves, but avoids paying the mesh’s connection/resource tax.
What we’re trying to figure out: is there a concrete, measurable downside to the full-mesh approach (once flexibility is set aside), e.g. does per-shard sender/receiver overhead scale in a way that causes real contention at cluster counts like this (say 4 to 8 clusters), or is it negligible in practice? Has anyone run mesh topologies at this shard count/cluster count and hit resource or isolation issues that pushed them toward pairs instead? Any metrics, guidance, or war stories would help us decide whether moving to pairs is worth the flexibility we’d give up.
Thanks,
Hardy
Regarding the downsides of a full mesh, the concern you’ve identified is correct. Overhead scales with the number of cluster pairs. With n clusters you end up with n(n−1) directed connections; each one opens a sender/receiver stream per shard, producing roughly n(n−1) × 4096 streams. Those streams exist even when no namespaces are actively replicating on that pair, so you pay for idle goroutines, connection state, monitoring, bookkeeping, etc. In addition, a problem on any single link or cluster can create side effects across the mesh through shared resources on the history hosts (network, CPU, connection management, etc.).
I would likely opt to start with the simpler approach of establishing replication pairs only as needed. If the lack of full-mesh flexibility later becomes a significant shortcoming, then you can always expand the topology. I would plan your resources carefully in any event (i.e. plan your initialFailoverVersion/failoverVersionIncrement ahead of time) in order to avoid any potential issues with expanding later on.
Thanks @dspinhirne for confirming this is a real concern, and the n(n−1) shard-stream math matches what I found in the code too.
I dug into the implementation a bit more (caveat: this is from reading the source, not runtime measurements)
- Goroutines/connections themselves probably aren’t the bottleneck. Go can handle far more go routines than a mesh this size produces, and idle streams cost little memory.
- Replication traffic sender side looks well isolated. Each stream has its own rate limiter and notification channel, so a slow target doesn’t throttle others.
- Receive side is more nuanced. Replication task application runs through a worker pool shared per host across all replication streams. An idle stream costs nothing here, but if a cluster becomes the active target of multiple concurrently-replicating sources (“fan-in”), those sources compete for the same pool. Pairs cap fan-in at 1 by construction; mesh makes it possible, but only actually bites if you route several active namespaces into the same cluster at once. Worth noting this is tunable, not a hard ceiling.
ReplicationProcessorSchedulerWorkerCount (and its low-priority counterpart) control the pool size, so it’s something you can size for rather than just hope doesn’t happen.
- Persistence QPS is the other real cost, and it’s source-side only. Task generation is gated per-namespace (multi-cluster or not), not per-target-cluster — so with 4 meshed clusters A–D and a namespace replicating only [A,B,C], cluster A still wakes, reads, and discards on behalf of its sender to D. No task data crosses to D, though a watermark-only ping can eventually go out to keep D’s ack tracking alive.
The appeal of mesh for us is better namespace-placement options. Choose the cluster with the most spare capacity up front, then later replicate toward whichever cluster has capacity at that time, rather than being locked to a fixed partner.
All of the above is code-reading, not measurement, though. I don’t have real production data to back it up. Would love to hear if anyone has actual numbers from running a mesh at scale, and especially would appreciate it if any Temporal maintainers could confirm or correct the mechanisms above.
--Hardy