We are able to deploy Temporal cluster to kubernetes clusters, however the membership ring (of history service and matcher service) can only be configured to work within the same kubernete cluster. The membership communication seems to be rely on the broadcastAddress configured (Temporal Cluster configuration reference | Temporal Documentation) and it only allows IP address.
We would prefer strong availability over multiple kubernetes clusters. In particular, we would have a distributed database (e.g. Cassandra) cluster deployed across DC, and have Temporal cluster deployed on different kubernete clusters across DC but using same Cassandra cluster, thereby achieving cross-DC availability.
Is there any plan to support above mentioned features?
The latency is around 200ms. We can look at different deployment for different use cases. For those require high performance, we can deploy within same kubernetes cluster and use global namespace for replication. For those use case with high availability requirement but not much expectation on latency, we would use cross dc deployment.
Would also suggest Temporal can support both above modes of deployment .
I doubt that with 200 ms you would be able to get this running with any reasonable performance. And you still need to make sure that all temporal service hosts can talk to each other directly.
Yeah, the performance would be slow. We are still trying it out, however for the current use cases, the workflow likely to need days to run. A few seconds delay at each step would be acceptable compared the availability benefit we get.
I can deploy all the service instance such that each has its own URL, in a format like <temporal_cluster_id><kubernets_cluster_id><temporal_service_name>_<temporal_service_instance_id>:<port_number>. I can create kubernetes objects constructs needed for creating the URLs. However the chanllenge now is the ringpop implmeentation appears only accept Host:Port format. Can this configuration format be changed to accept URL format?
We have tried suggestions from Derek, most of these require kubernetes admin privilege, e.g. calico, or vpn, however at our firm, dev teams do not have such access level at infra.
We have figured how to deploy Temporal across GKP cluster with envoy. However would like to ask another question, what is the consistency levels and how to configure it for Temporal’s microservice (e.g. History or matcher).
.