Tuning Temporal setup for better performance

Community Support
, ,
5

Main parameters to tune

Defining pooling the tasks from the server:

WorkerOptions#workflowPollThreadCount

WorkerOptions#activityPollThreadCount

WorkerOptions#maxConcurrentWorkflowTaskExecutionSize

WorkerOptions#maxConcurrentActivityExecutionSize

Defining the in-memory cached workflows state and threads:

WorkerFactoryOptions#maxWorkflowThreadCount

WorkerFactoryOptions#workflowCacheSize

Some reasonable limitations for these values

  1. WorkerFactoryOptions#workflowCacheSize ≤ WorkerFactoryOptions#maxWorkflowThreadCount. Having a cache larger than the size of the thread pool doesn’t make much sense.
  2. WorkerOptions#maxConcurrentWorkflowTaskExecutionSize ≤ WorkerFactoryOptions#maxWorkflowThreadCount. maxWorkflowThreadCount should be ideally at least 2x of maxConcurrentWorkflowTaskExecutionSize to be safe, but it depends on how actively an app uses threads and also how active the workflows are. Having maxConcurrentWorkflowTaskExecutionSize > maxWorkflowThreadCount doesn’t make sense at all and it’s a misconfiguration because for each workflow only one Workflow Task can be processed at a single moment of the time anyway.

The desired poller count

depends on

  1. The latency from the worker to the service. Higher latency lowers the throughput of a single poller thread.
  2. If you see that worker threads are not getting loaded with enough job, at the same time schedule_to_start latencies are high, you can try increasing the poller count to 5-10 pollers.
  3. How big are the workflow tasks (how large is the average amount of time between consecutive blockage of workflow execution on activities or waiting/sleeping)? Smaller workflow tasks your application has, larger pollers/executors ratio you need.

The desired executors count

depends on the utilization of resources of your worker. If the worker is underutilized, increase maxConcurrent*ExecutionSize.

Workflow tasks

It doesn’t make much sense to set WorkerOptions#maxConcurrentWorkflowTaskExecutionSize value too high. Because Workflow code shouldn’t have blocking operations and waits [other than the ones provided by Workflow class], Workflow Tasks should occupy a full core during execution. This means that it doesn’t make much sense to set WorkerOptions#maxConcurrentWorkflowTaskExecutionSize into something much higher than the amount of the cores.

Activities

WorkerOptions#maxConcurrentActivityExecutionSize should be set looking into the profile of your Activities. If the Activities are mostly computational, it doesn’t make much sense to set it into something larger than a number of available cores. But if Activities perform mostly input-output awaiting RPC calls, it makes sense to increase this number by a lot.

Drawbacks of putting “very large values”

As with any multithreading system, specifying too large values will lead to too many active threads performing work and constant resources stealing and switching, which will decrease the total throughput and latency jitter of the system.

1 Like