# RESOURCE\_EXHAUSTED: Too many outstanding requests to the service **URL:** \ **Category:** Community Support\ **Created:** [May 17, 2021, 7:49pm UTC](https://community.temporal.io/t/resource-exhausted-too-many-outstanding-requests-to-the-service/2158 "2021-05-17T19:49:01Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![arjunraman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/arjunraman/32/865_2.png) [@arjunraman](https://community.temporal.io/u/arjunraman)\ **Post date:** [May 17, 2021, 7:49pm UTC](https://community.temporal.io/t/resource-exhausted-too-many-outstanding-requests-to-the-service/2158/1 "2021-05-17T19:49:01Z")
Hello, We are load testing a simple workflow with 1 activity (that prints to console) on a single Aurora MySQL db to figure out the limits of system. At high QPS, I’m starting to see the below errors in the log and wanted to find out how I can scale further. The db is at 50% CPU, so definitely has a lot of headroom left. Stack trace: {"@timestamp":“2021-05-14T08:28:39.799+00:00”,“trace\_id”:“9mgL46qlHwuaJkkHfpSe\_w==”,“span\_id”:“asaGgrywBI4tBd1pQCy27w==”,“level”:“WARN”,“level\_value”:30000,“logger\_name”:“io.temporal.internal.common.GrpcRetryer”,“thread\_name”:“pool-16-thread-53”,“exception”:{“class”:“io.grpc.StatusRuntimeException”,“message”:“RESOURCE\_EXHAUSTED: Too many outstanding requests to the service.”},“fingerprint”:“5bf31268”,“message”:“Retrying after failure”} ! io.grpc.StatusRuntimeException: RESOURCE\_EXHAUSTED: Too many outstanding requests to the service. ! at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ! at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ! at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ! at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.startWorkflowExecution(WorkflowServiceGrpc.java:2614) ! at io.temporal.internal.external.GenericWorkflowClientExternalImpl.lambda$start$0(GenericWorkflowClientExternalImpl.java:88) ! at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:97) ! at io.temporal.internal.external.GenericWorkflowClientExternalImpl.start(GenericWorkflowClientExternalImpl.java:81) ! at io.temporal.internal.sync.WorkflowStubImpl.startWithOptions(WorkflowStubImpl.java:155) ! at io.temporal.internal.sync.WorkflowStubImpl.start(WorkflowStubImpl.java:268) ! at io.temporal.internal.sync.WorkflowInvocationHandler$StartWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:245) ! at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:181) ! at com.sun.proxy.$Proxy130.execute(Unknown Source) ! at io.temporal.internal.sync.WorkflowClientInternal.start(WorkflowClientInternal.java:218) ! at io.temporal.client.WorkflowClient.start(WorkflowClient.java:238) ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 17, 2021, 8:07pm UTC](https://community.temporal.io/t/resource-exhausted-too-many-outstanding-requests-to-the-service/2158/2 "2021-05-17T20:07:48Z")
How many shards have you configured for this cluster? How many nodes of each role do you have? What is these nodes CPU utlization? ---
**Author:** ![arjunraman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/arjunraman/32/865_2.png) [@arjunraman](https://community.temporal.io/u/arjunraman)\ **Post date:** [May 17, 2021, 8:55pm UTC](https://community.temporal.io/t/resource-exhausted-too-many-outstanding-requests-to-the-service/2158/3 "2021-05-17T20:55:32Z")
> How many shards have you configured for this cluster? 512 > How many nodes of each role do you have? temporal web: 6 pods temporal frontend: 12 pods temporal history: 30 pods temporal matching: 9 pods temporal worker: 9 pods > What is these nodes CPU utlization? Hard to exactly say given that we are running on top of K8s. Initially when we started the load test, we were breaching the CPU limit of cores. Increased the history server to 30 pods and now we briefly exceed the CPU limit of 4 cores, but not as bad as before ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [May 17, 2021, 9:11pm UTC](https://community.temporal.io/t/resource-exhausted-too-many-outstanding-requests-to-the-service/2158/4 "2021-05-17T21:11:25Z")
> try to increase the following dynamic config to higher value ```auto frontend.rps frontend.namespaceRPS frontend.namespaceCount ``` ---
**Author:** ![arjunraman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/arjunraman/32/865_2.png) [@arjunraman](https://community.temporal.io/u/arjunraman)\ **Post date:** [May 17, 2021, 9:39pm UTC](https://community.temporal.io/t/resource-exhausted-too-many-outstanding-requests-to-the-service/2158/5 "2021-05-17T21:39:49Z")
Thanks @Wenquan_Xing ! Quick question, what are the default values for: ``` frontend.rps frontend.namespaceRPS frontend.namespaceCount ``` ---
**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\ **Post date:** [May 17, 2021, 9:45pm UTC](https://community.temporal.io/t/resource-exhausted-too-many-outstanding-requests-to-the-service/2158/6 "2021-05-17T21:45:48Z")
>