Hi,
Getting error of this kind: RESOURCE_EXHAUSTED: namespace count limit exceeded
We have 4 workflows, 12 activities and 5 local activities. Value for maxConcurrentWorkflowTaskPollers is defined as 80 for each workflow and maxConcurrentActivityTaskPollers is defined as 50 for each poller. As per my understanding, default value of frontend.namespaceCount is 1200, but still getting error for the above configuration.
Calculation with above configuration should be like 4 * 80 + 12 * 50 = 920.
Can someone please explain the calculation behind this count - frontend.namespaceCount?
Local activities also need to be considered as part of this ?
1 Like
RESOURCE_EXHAUSTED: namespace count limit exceeded
Too many concurrent polls for workflow/activity tasks to a single frontend host. The default frontend.namespaceCount is 1200 and you probably need to increase it.
How much to increase, would check service_pending_requests gauge metric which tracks total number of outstanding polls and would tell you how many total concurrent polls server is seeing. (note this metric was added in server version v1.14.0)
2 Likes
Should service_pending_requests be as close to 0 as possible for each namespace or is it expected that a namespace with a lot of activity would also have a high number of pending requests? I just noticed this metric, and see one of our namespaces with a very high pending request count, but we’re also not seeing any RESOURCE_EXHAUSTED errors.
service_pending_requests is emitted when there is a poller polling on frontend service and waiting for a task. It should not be 0 as that would mean there are no pollers.
High number is not guaranteed to be a bad sign, it could just mean you have a bunch of pollers “doing nothing”. I believe you should start seeing resource exhausted logs for a namespace if this service_pending_requests reaches its frontend.namespaceCount.
2 Likes
It’s not possible to update server dynamic config from SDK apis. You can specify dynamicConfigClient pollInterval in static config to define how often values in your dynamic config should be updated.
What’s the use case for needing to do it from SDK/client side?
1 Like
Hi Tihomir, Thanks again, I actually moved the question to the following question How to add new PollerOption to workflow? - #5 by Sachin_Panemangalore , since it has context around the reason for the change.