Temporal server errors

Community Support
,
1

Hi all,

We use temporal self hosted server for our business. Recently we have been getting the below mentioned errors intermittently. We are not able to find the resolution or the root cause for these issues. One of the observation we had was load was also increased during this period.

Errors

  1. Error1 call cancelled
  2. Error: 14 Unavailable: ListWorkflowExecutions failed: elastic: Error 503(service unavailable)
  3. Error: 8 RESOURCE_EXHAUSTED: namespace rate limit exceeded

Can someone help us out in finding the root cause or the resolutions for the above mentioned errors?
TIA:)

2

Dynamic configs:

frontend.namespaceRPS.visibility (default 10)
frontend.namespaceBurstRatio.visibility (default 1)
frontend.globalNamespaceRPS.visibility (default 0 meaning no limit)

You should just have to up frontend.namespareRps from default.
Server emits resource exhausted info as well via metrics, Grafana query

sum(rate(service_errors_resource_exhausted{}[1m])) by (operation, resource_exhausted_cause)