Getting error of this kind: RESOURCE_EXHAUSTED: namespace count limit exceeded
We have 4 workflows, 12 activities and 5 local activities. Value for maxConcurrentWorkflowTaskPollers is defined as 80 for each workflow and maxConcurrentActivityTaskPollers is defined as 50 for each poller. As per my understanding, default value of frontend.namespaceCount is 1200, but still getting error for the above configuration.
Calculation with above configuration should be like 4 * 80 + 12 * 50 = 920.
Can someone please explain the calculation behind this count - frontend.namespaceCount?
Local activities also need to be considered as part of this ?
RESOURCE_EXHAUSTED: namespace count limit exceeded
Too many concurrent polls for workflow/activity tasks to a single frontend host. The default frontend.namespaceCount is 1200 and you probably need to increase it.
How much to increase, would check
service_pending_requests gauge metric which tracks total number of outstanding polls and would tell you how many total concurrent polls server is seeing. (note this metric was added in server version v1.14.0)
service_pending_requests be as close to 0 as possible for each namespace or is it expected that a namespace with a lot of activity would also have a high number of pending requests? I just noticed this metric, and see one of our namespaces with a very high pending request count, but we’re also not seeing any
service_pending_requests is emitted when there is a poller polling on frontend service and waiting for a task. It should not be 0 as that would mean there are no pollers.
High number is not guaranteed to be a bad sign, it could just mean you have a bunch of pollers “doing nothing”. I believe you should start seeing resource exhausted logs for a namespace if this service_pending_requests reaches its frontend.namespaceCount.