Independent Temporal clusters with a shared data layer?

Community Support Server Deployment
1

We’re working on a “cell” architecture where no cell knows about any other (so they can go offline and the remaining cells are unaffected) where each cell has a Temporal cluster in k8s, but all cells share a global aurora DB. So every Temporal instances sees every workflow in every cell, since the datalayer is shared. This works currently in that every Temporal UI shows every workflow across every cluster/AZ/region just as expected.

There is no connectivity between cells though so no Temporal cluster can gossip with any other so they all share a cluster_membership table and all have the same membership_partition which appears hardcoded to 0: temporal/common/persistence/sql/sqlplugin/postgresql/cluster_metadata.go at main · temporalio/temporal · GitHub

I’m curious if Temporal can run in this fashion, for instance Keycloak uses a JDBC_PING column so you can have independent clusters but a shared data layer.

Having the isolated Temporal clusters with a shared data-layer seems to cause some shard ownership churning, and timeouts while trying to connect to IPs of Temporal pods that belong to different k8s clusters.

2

This is not going to work as a single DB cannot be shared across multiple Temporal clusters. You can run a single cluster across multiple k8s clusters, but your networking should be configured to support direct point to point connections between any two Temporal nodes.