Resolve namespace concurrent poller limit exceeded using tctl

Community Support
1

Hi,

I am experiencing a consistent flow of warning into logs “Error namespace concurrent poller limit exceeded”.

I did a some digging on this forums and it seems like I have to do two things
check service_pending_requests and increase frontend.namespaceCount. The temporal server is deployed on a private EKS cluster.

I am trying to figure out where can I see those metrics and how can I set frontend.namespacecount using tctl(we don’t have temporal cli installed).

2

Hi Mickey,

You can query service_pending_requests metrics from either Prometheus or Grafana assuming you had them setup. Otherwise, this guide should help you to get started.

To increase frontend.namespaceCount, you need to know where your dynamic config yaml file is located (tctl won’t work). The default location is /etc/temporal/config/dynamicconfig/docker.yaml but it’s commonly configured via DYNAMIC_CONFIG_FILE_PATH env var on the server side. If you happen to use our helm-chart, here is the config instruction.

frontend.namespaceCount comes with the default value of 1200. Below is a snippet of the dynamic config yaml that shows the bumped value:

frontend.namespaceCount:
  - value: 4800
    constraints: {}

While increasing concurrency threshold on the server side is a common approach, you may also want to consider limiting the concurrency level on the worker side to prevent them from DDoSing any downstream services during traffic spike. This is a good read on worker tuning. You may find the configurations section with parameters, such as maxConcurrentActivityExecutionSize, setMaxConcurrentActivityTaskPollers, particularly useful for reducing load on the server.

Hope this helps.

References:
- Developer's guide - Worker performance | Temporal Documentation
- How to add new PollerOption to workflow? - #6 by tihomir
- How to specify dynamic config

1 Like