How to resolve RESOURCE_EXHAUSTED: namespace count limit exceeded

Community Support
1

Temporal v1.10.5

All of our workers connected to one particular namespace are throwing the following exception:

Failure in thread Workflow Poller taskQueue="the-tq", namespace="blah": 3
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: namespace count limit exceeded
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollWorkflowTaskQueue(WorkflowServiceGrpc.java:2652)
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:81)
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:37)
	at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:283)
	at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:242)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

It’s unclear to me what this means, or what the resolution might be. It doesn’t appear that our workers’ are degraded in any significant way, but it just started happening out of the blue and hasn’t recovered since.

2

the above message means there are too many concurrent poll for workflow task / poll for activity task to a single frontend host

you can override the default frontend.namespaceCount from 1200 to higher number

2 Likes