# Some deployment questions for a redundant self-hosted deployment **URL:** \ **Category:** Community Support\ **Tags:** general-impl\ **Created:** [May 7, 2021, 10:17am UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096 "2021-05-07T10:17:00Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![horre](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@horre](https://community.temporal.io/u/horre)\ **Post date:** [May 7, 2021, 10:17am UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/1 "2021-05-07T10:17:00Z")
We are currently developing a service that runs a REST API and on certain request needs to trigger long running external services. So for example if a user registers, we need to initialise certain other services that might take a couple of minutes. So we are looking into Temporal to execute those as workflows. For the deployment we currently have 3 large servers in our own datacenter (dom1, dom2, dom3) that run multiple services in linux containers. Each of the doms runs a MariaDB instance (mdb1, mdb2, mdb3) that are synchronised as a Galera cluster. And each of the doms also runs an api node (api1, api2, api3) to serve the requests. To make sure that we can survive both dom failure and single container failure, a load balancer distributes requests among the healthy api nodes and each api node is connected to all 3 database nodes. We are not expecting high throughput on the workflows and executing those is not very time critical, but it’s important for us that workflows are started reliably. So what we would like is a cluster of Temporal servers that use our Galera cluster as backend. So on each dom, we would like to deploy one frontend, matching, history and worker service for starters (and eventually scale up services later if needed). Temporal services should be connected with Temporal servers on the other doms and each Temporal service should be connected to all 3 database nodes. Each api node should then utilise all 3 Temporal frontends to start the workflows. Does this deployment make sense (or would it even work like this)? Can Temporal utilize a MariaDB Galera cluster as datastore (can we just deploy the servers with multiple persistent sql connections)? And is there a way to utilize the SDK client to connect to multiple Temporal servers for starting workflows? Or would we need to handle those cases ourselves to try a different Temporal server on failure? ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 7, 2021, 7:13pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/2 "2021-05-07T19:13:17Z")
Temporal required strong consistency from a DB. Does the MariaDB Galera cluster provide such a guarantee? If it loses some events during failover it might lead to data corruption. ---
**Author:** ![horre](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@horre](https://community.temporal.io/u/horre)\ **Post date:** [May 10, 2021, 11:34am UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/3 "2021-05-10T11:34:10Z")
Yes Galera clusters do provide strong consistency. ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 10, 2021, 3:59pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/4 "2021-05-10T15:59:10Z")
Then your setup should work. I think if you care about reliability having just one instance of each role is not enough as anytime any of the roles goes down the service will be unavailable until it is restarted. > And is there a way to utilize the SDK client to connect to multiple Temporal servers for starting workflows? SDK client connects to multiple instances of frontends but not multiple instances of the service. The single service instance can be pretty highly available if deployed correctly. ---
**Author:** ![horre](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@horre](https://community.temporal.io/u/horre)\ **Post date:** [May 10, 2021, 4:22pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/5 "2021-05-10T16:22:28Z")
We will have all roles (I guess this means frontend, worker, history and matching service?) deployed on all dom servers, so for each role there will be 3 nodes running. > SDK client connects to multiple instances of frontends but not multiple instances of the service. The single service instance can be pretty highly available if deployed correctly. Can I specify which service the client is connected to or is this handled by the frontend? E.g. can I have 3 clients initialised on my API node that are connected to different service nodes and if client1 fails, try client2? ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 10, 2021, 4:50pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/6 "2021-05-10T16:50:32Z")
You mentioned that you already have a load balancer in front of the frontend nodes. So clients will reconnect to whatever frontend is available. ---
**Author:** ![horre](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@horre](https://community.temporal.io/u/horre)\ **Post date:** [May 10, 2021, 5:00pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/7 "2021-05-10T17:00:15Z")
Ah okay now I understand. Yeah the load balancer was only for external requests to our API nodes. But we can setup an internal one to connect our API nodes to the Temporal services as well. ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 10, 2021, 5:15pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/8 "2021-05-10T17:15:36Z")
Temporal gRPC client will load balance [using round\_robin policy](https://github.com/temporalio/sdk-java/blob/ba691a245bfb4fd8450f4c620cbeaff130917076/temporal-serviceclient/src/main/java/io/temporal/serviceclient/WorkflowServiceStubsImpl.java#L113) by default so there is no hard requirement to have a load balancer in the middle. ---
**Author:** ![horre](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@horre](https://community.temporal.io/u/horre)\ **Post date:** [May 11, 2021, 12:37pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/9 "2021-05-11T12:37:22Z")
We are using the Go client which seems to only have a single HostPort connection option (at least for the initial connection), see [here](https://github.com/temporalio/sdk-go/blob/0ef35cb9e668dea909c46ea89c3ad42413c5c3be/internal/client.go#L334). But as I understand the grpc client will hold a connection pool after the initial dial up? So we would set up a load balancer for the frontend service, use the HostPort of that load balancer for the initial client setup (so we are sure to initially connect) and afterwards the client will hold a connection pool and will handle load balancing by itself? ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 11, 2021, 3:10pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/10 "2021-05-11T15:10:15Z")
I’m not a networking expert, but I think if a load balancer is used then all the connections go through it. [This](https://stackoverflow.com/a/63800475/1664318) can also help in some cases. ---
**Author:** ![horre](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@horre](https://community.temporal.io/u/horre)\ **Post date:** [May 11, 2021, 3:15pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/11 "2021-05-11T15:15:25Z")
Ah no it’s just a dns based assignment. So it’s basically `dns:///temporal-frontend-load-balancer` as `HostPort` config, where the load balancer would then return an ip address from a healthy temporal frontend service. The client then connects to the actual temporal service directly. ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 11, 2021, 3:30pm UTC](https://community.temporal.io/t/some-deployment-questions-for-a-redundant-self-hosted-deployment/2096/12 "2021-05-11T15:30:41Z")
I see. The dns based assignment does make sense.