Workflow Queries Tab - upstream connect error or disconnect/reset before headers. reset reason: overflow


Temporal Server: v1.18.3
Temporal UI: 2.8.0
Platform: Kubernetes

  1. Which Temporal service will be invoke when I click on Workflow–> Queries tab for custom search attributes (GetSearchAttribues) ?

  2. Is there a limit of Temporal Service pods to be run in k8s for each service?

  3. Can we have all the Temporal Service pods in equal scale?

  4. Which Temporal service consumes more cpu & memory (Resource Quota)?

1: All clients requests go through frontend service
2, 3:. You can scale each service independently. Typically there is no need to scale up worker service.

Frontend service is mostly forwarding requests to history/matching (so CPU heavy, especially when needs to read and return event history to workers, meaning optimizing worker cache can help).

Matching service is mostly CPU heavy (assuming you have good sync-match rate, meaning optimized workers for the workload, otherwise it can also be DB heavy if sync match is low).

History service can be both cpu and memory heavy especially when you have large workloads. And is db heavy.