RateLimit issue in DescribeTaskQueue java-sdk-1.25.1 and Temporal-server-1.24.2

Community Support
,
1

Hi @tihomir,

I have recently upgraded our temporal server from temporal 1.22.2 to 1.24.2 and after upgrade only the describeTaskQueue has started throwing rate limit exception when below piece of code is run by multiple threads.
I’m able to reproduce issue with just 2 threads in Apache Jmeter also.
Same piece of code is able to handle the load of multiple threads in temporal server 1.22.2

DescribeTaskQueueResponse wfResponse = service.blockingStub().describeTaskQueue(taskQrqst);

Above code throws below exception since temporal server 1.24.2.
I tried temporal server 1.25.0 as well there as well same exception reproduces.

io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: namespace rate limit exceeded
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.describeTaskQueue(WorkflowServiceGrpc.java:4739)

To reproduce the issue locally I have created a sample code in below github repo.

You can find a method testWithConcurrency() in the class:
springboot/src/test/java/io/temporal/samples/springboot/HelloSampleTest.java

Run that to test to reproduce the rate limit exception in running temporal server 1.24.2 or 1.25.0 anyone.
I’ve configured test to run just 10 threads and run same method testGetQueueStatus().

With temporal server 1.22.2 there’s no exception of rate limit and test works fine.

New code to reproduce rate-limit bug in latest temporal server by ShubhSingh · Pull Request #1 · ShubhSingh/samples-java (github.com)

Note: The reason we are doing describeTaskQueue is to check the health of Task queues in production.
And since health check is frequently keeps happening describeTaskQueue rate limit is making health checks break.

2

In server release 1.24 DescribeTaskQueue api was changed to be bound by visibility rps limits.
If you need high rate of invocation of this api you need to update

frontend.namespaceRPS.visibility:
  - value: XYZ
    constraints: {}

in your dynamic config

1 Like
3

Thanks @tihomir this worked I have tested in local and our dev environment both.