Frontend - namespaceCount attribute


Getting error of this kind: RESOURCE_EXHAUSTED: namespace count limit exceeded

We have 4 workflows, 12 activities and 5 local activities. Value for maxConcurrentWorkflowTaskPollers is defined as 80 for each workflow and maxConcurrentActivityTaskPollers is defined as 50 for each poller. As per my understanding, default value of frontend.namespaceCount is 1200, but still getting error for the above configuration.

Calculation with above configuration should be like 4 * 80 + 12 * 50 = 920.
Can someone please explain the calculation behind this count - frontend.namespaceCount?
Local activities also need to be considered as part of this ?

1 Like

RESOURCE_EXHAUSTED: namespace count limit exceeded

Too many concurrent polls for workflow/activity tasks to a single frontend host. The default frontend.namespaceCount is 1200 and you probably need to increase it.

How much to increase, would check service_pending_requests gauge metric which tracks total number of outstanding polls and would tell you how many total concurrent polls server is seeing. (note this metric was added in server version v1.14.0)

1 Like

Should service_pending_requests be as close to 0 as possible for each namespace or is it expected that a namespace with a lot of activity would also have a high number of pending requests? I just noticed this metric, and see one of our namespaces with a very high pending request count, but we’re also not seeing any RESOURCE_EXHAUSTED errors.

service_pending_requests is emitted when there is a poller polling on frontend service and waiting for a task. It should not be 0 as that would mean there are no pollers.
High number is not guaranteed to be a bad sign, it could just mean you have a bunch of pollers “doing nothing”. I believe you should start seeing resource exhausted logs for a namespace if this service_pending_requests reaches its frontend.namespaceCount.

1 Like