Prioritizing different workers using fairnessKey and priorityKey in Java SDK

Community Support
, ,
1

Is fairnessKey and priorityKey in io.temporal.common.Priority in Java-SDK available for use with open-source Temporal, if so which temporal server version supports it ?

When should we consider using fairnessKey over priorityKey and vice-versa ?

We have 2 different kinds of workers -

  1. HighPriority Worker - Stable traffic throughout the day but lower TPS
  2. LowPriority Worker - Burst traffic at particular times in a day but TPS is very high

We don’t want our HighPriority Worker get impacted because of the burst traffic of the LowPriority Worker. We don’t want HighPriority taskQueue’s stable traffic starve for Worker/resources when the burst traffic arrives. We can throttle the processing of the LowPriority Worker and it can be executed slowly.

We tried configuring the following WorkerOptions properties for the 2 workers. For HighPriority TaskQueue/Workers we set the following properties to a higher values compared to the LowPriority TaskQueue/Workers -

setMaxConcurrentActivityExecutionSize
setMaxConcurrentWorkflowTaskExecutionSize

Setting the following WorkerOptions properties to PollerBehaviorAutoscaling

setWorkflowTaskPollersBehavior
setActivityTaskPollersBehavior

But still we see during burst traffic on LowPriority taskQueue, execution of HighPriority taskQueue slows down.

Any guidance/pointers in this regard is appreciated. Thanks !

2

Hi

Is fairnessKey and priorityKey in io.temporal.common.Priority in Java-SDK available for use with open-source Temporal, if so which temporal server version supports it ?

yes it is, for the java sdk, check the release noter for Release v1.29.0 · temporalio/sdk-java · GitHub for priority and Release v1.31.0 · temporalio/sdk-java · GitHub for fairness

Temporal server

In your case:
If you want to guarantee that HighPriority tasks are never starved by bursts of LowPriority tasks you can use priorityKey.

If you also want to control the share of resources among multiple tenants or groups within the same priority, then you could additionally use fairnessKey and fairnessWeight.

As of now, both features are experimental and not yet GA in OSS Temporal.

Priority in Temporal is strict within a single task queue partition, but not globally strict across all partitions. There can be rare cases where a low priority task is dispatched before a hight priority tasks. If you need strict priority you can set the task queue partition to 1.

3

Priority in Temporal is strict within a single task queue partition, but not globally strict across all partitions. There can be rare cases where a low priority task is dispatched before a hight priority tasks. If you need strict priority you can set the task queue partition to 1.

@antonio.perez - Thanks for the detailed response !

So does that mean if my taskQueues are partitioned (which are partitioned by default), the setPriority doesn’t guarantee strictness on priority always.

Also setting the taskQueue partition to 1 would mean reduced rate of polling and execution and overall slowness and reduced throughput ?