Need Help in Mapping method call with Rate limit property

Community Support
1

Hi Temporal Team

We extensively use methods in WorkflowServiceGrpc class for monitoring workflows. In the process we at times encounter exception: RESOURCE_EXHAUSTED: namespace rate limit exceeded

Need your inputs regarding which rate limit property value should be changed.

I have looked at the following threads:

RESOURCE_EXHAUSTED: namespace rate limit exceeded - Community Support - Temporal

Wondering if changing the following RPS values will address the issue:

frontend.rps:

frontend.namespaceRPS:

temporal/constants.go at master · temporalio/temporal · GitHub

Class WorkflowServiceGrpc,

Methods called frequently.:

public static MethodDescriptor<GetClusterInfoRequest, GetClusterInfoResponse> getGetClusterInfoMethod()

public ListNamespacesResponse listNamespaces(ListNamespacesRequest request)

public ListenableFuture listOpenWorkflowExecutions(ListOpenWorkflowExecutionsRequest request)

Thanks in advance.
Best Regards
Krishnendu

2

Hi @tihomir Any pointers… Thank you.

3

RESOURCE_EXHAUSTED: namespace rate limit exceeded

Would check server metrics as well:

sum(rate(service_errors_resource_exhausted{}[1m])) by (operation, resource_exhausted_cause)

For RPS limits it could be due to per-namespace RPS on this cluster or overall RPS limit so dynamic config as you mentioned:

frontend.rps for frontend overall rps limit

frontend.namespaceRPS for frontend per namespace rps limit.

Note that there is also visibility api rps limits and associated dynamic configs:

frontend.namespaceRPS.visibility (default is set to 10 just fyi)

Would use server metrics to find out which apis (operations) you are getting rate limited on and then go from there.

4

Thank you @tihomir :slight_smile:

5

Hi @tihomir

We are getting the exception while calling the following methods (note below), all the same its intermittent. I tried replicating the issue by changing the values in dynamic config:

frontend.rps:

  • value: 1
    constraints: {}
    frontend.namespaceRPS:
  • value: 1
    constraints: {}
    frontend.namespaceBurst:
  • value: 1
    constraints: {}

However even with the above changes the issue did not replicate immediately. Is there a default minimum value for the above constants which will override if we provide small values from our end.

Please suggest.
Thank you
Best Regards
Krishnendu

Class–>> WorkflowServiceGrpc

public static MethodDescriptor<GetClusterInfoRequest, GetClusterInfoResponse> getGetClusterInfoMethod()

public ListNamespacesResponse listNamespaces(ListNamespacesRequest request)

public ListenableFuture listOpenWorkflowExecutions(ListOpenWorkflowExecutionsRequest request)

6

Hi @tihomir

Any inputs as to why I am not getting an exception when I set frontend.rps and frontend.namespaceRPS values to 1 in dymanicconfig/development.yaml

teporalrps
teporalrps2542×1246 354 KB
.

Basically, I am setting the values or combination incorrectly.

Thanks again for all the help…
Best Regards
Krishnendu

7

How many frontend instances do you have? frontend.namespaceRPS is per-host setting. Can you give bit more info on your deployment?

8

Hi @tihomir I have created 100 namespaces and set the frontend.namespaceRPS value to 1;

And ran workflows in order to replicate the issue RESOURCE_EXHAUSTED: namespace rate limit exceeded.

But even with 1 value the issue is seen very sparsely in my local VM. However the issue is seen frequently in similar environment in Kubernetes cluster.

Just wondering if the exception has something to do regarding deployment hardware/ software.

Thanks again
Regards
Krishnendu

9

Hi @tihomir

I was working with an older version of Temporal, migrated my setup to current version and the issue replicated immediately with frontend.rps & frontend.namespaceRPS value to set 1; for one namespace.

I was able to resolve the issue by changing these values to higher number.

We can close this thread…

Thanks in advance.
Best Regards
Krishnendu