I am getting an error “observation: RESOURCE_EXHAUSTED: namespace concurrent poller limit exceeded” . Would the above configurations work in this case ?
Also, in the Java SDK Client , the above methods “setMaxConcurrentWorkflow/ActivityTaskPollers” don’t seem to be available , has this configuration changed ?
Thanks Tihomir, we are currently using Temporal Java SDK 1.16.0 . A few related questions.
Q1. To reduce the number of pollers/workers polling on this namespace - can I use the setMaxWorkflowThreadCount() API ? - This is currently set to the default value of 600 - does this still need to be reduced ?
/** Java SDK **/
private static final int DEFAULT_HOST_LOCAL_WORKFLOW_POLL_THREAD_COUNT = 5;
private static final int DEFAULT_WORKFLOW_CACHE_SIZE = 600;
private static final int DEFAULT_MAX_WORKFLOW_THREAD_COUNT = 600;
Q2. frontend.namespaceCount - What is this counter used for ? is there a way to calculate the appropriate lower-bound for this w.r.t number of active workflows ? Also, is there a way to update this value using the Temporal SDK ?
Q3. We currently have a single pod running Temporal frontend. Should the number of pods for Temporal frontend be based on the mean CPU utilization of the frontend pod ?
For poller counts you would look at WorkerOptions->setMaxConcurrentWorkflowTaskPollers (default 5) and WorkerOptions->setMaxConcurrentActivityTaskPollers (default 5) WorkerFactoryOptions->setMaxWorkflowThreadCount is setting for max number of actual threads available to be used for workflow executions across all workers created from a particular factory.
frontend.namespaceCount can be configured in dynamic config (default 1200). It can be filtered by ns and is typically looked at when you see resource exhausted issues for example:
meaning there are too many concurrent polls for workflow/activity tasks to a single frontend host. So it has to do more with the number of workers, pollers and number of frontend hosts, rather than number of workflow executions (per namespace). No, dynamic config values cannot be updated from SDK.
Temporal server frontend service is for the large part responsible to route requests to history and matching service, and read the workflow event history for your sdk workers, so it is CPU heavy, yes.