Traffic is not load balanced among all worker-service instances deployed in Kubernetes

Community Support Server Deployment
1

Hi everyone,

We recently deployed Temporal on Kubernetes, with each service (frontend, history, matching, and worker) running in its own Deployment. Each service is configured with 2 replicas.

While observing the cluster, we noticed that only one replica of some services (the worker service in particular) appears to handle most of the traffic, while the second replica remains almost idle.

My understanding is that communication between Temporal services relies on gRPC. Since gRPC uses long-lived HTTP/2 connections, traffic is typically pinned to a single backend once a connection is established. As a result, Kubernetes Services alone may not provide effective load balancing across replicas unless additional mechanisms (service mesh, L7 load balancer, etc.) are introduced.

In our environment, deploying a service mesh is not an option. However, we do use Cilium, which can perform load balancing for ClusterIP Services through specific annotations, more details can be found here

The issue is that the Temporal Helm chart creates headless Services for inter-service communication. Because headless Services return pod IPs directly, traffic appears to remain pinned to a single replica.

This raises a few questions:

  1. What is the rationale behind using headless Services for Temporal’s internal services?
  2. Can these headless Services be safely replaced with regular ClusterIP Services?
  3. Is there a supported configuration (Helm chart or otherwise) to deploy ClusterIP Services instead of, or in addition to, the headless Services?
  4. If headless Services are required, shouldn’t the client-side gRPC implementation perform load balancing across the returned endpoints?

For example, something along these lines:

target := fmt.Sprintf("dns:///%s:%s", host, port)

dialOpts := []grpc.DialOption{
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
}

conn, err := grpc.NewClient(target, dialOpts...)
if err != nil {
    log.Fatalf("Failed to connect to gRPC server: %v", err)
}
defer conn.Close()

From what I can see, this would allow the gRPC client to resolve multiple pod IPs from the headless Service and distribute requests across them.

Am I misunderstanding how Temporal’s internal service discovery and load balancing work, or is there another mechanism that already handles this?

Thanks!

2

Hi @maxim , sorry to ping you directly.

I posted the following question a few days ago but haven’t received any responses yet. I realize it may not be your area of expertise, but I was wondering if you happen to have any ideas, or if you know someone on the Temporal team who might be able to take a look.

I’d really appreciate any pointers. Thanks!

3

@tihomir Sorry to bother you as well. Since this question hasn’t received any feedback yet, I was wondering if you might have any ideas on this, thanks!

4

My understanding is that communication between Temporal services relies on gRPC.

yes, transport is grpc, but Temporal intra-service communication is based on ringpop (membership discovery and routing). It does not rely on k8s or dns.

What is the rationale behind using headless Services for Temporal’s internal services?
this this is preferred actually so services can reach each others ports directly

Can these headless Services be safely replaced with regular ClusterIP Services?

for intra-service comms not needed
for frontend service incoming calls using clusterip/lb is typically expected

Is there a supported configuration (Helm chart or otherwise) to deploy ClusterIP Services instead of, or in addition to, the headless Services?

for intra-service comms, again, no

From what I can see, this would allow the gRPC client to resolve multiple pod IPs from the headless Service and distribute requests across them.

Temporal worker service clients connects to frontend via grpc with round_robin configured already. same if you dont configure publicClient.hostPort and have worker service role talk to internal-frontend using membership ring

While observing the cluster, we noticed that only one replica of some services (the worker service in particular) appears to handle most of the traffic, while the second replica remains almost idle.

i think this can be expected, for example for per-ns workers (for example workers created for schedules)
default dyn config is worker.perNamespaceWorkerCount = 1
if you have small num of namespaces having one worker pod that “does work” while other does very little or nothing is normal
you could increase perNamespaceWorkerCount if needed, but most likely you have small num of namespaces and maybe not using lots of schedules for example so dont need to scale out worker service yet