Communication between worker process and matching service

Community Support
1

A task queue can have partitions and each partition can be placed on a different instance of matching service.

Let’s say we have dozens of matching services. How does worker know which matching services are the owners of the task queues?

Is it recommended to auto scale matching services?
As task partitions will rebalanced if we autoscale, is there any performance implication?

2

Let’s say we have dozens of matching services. How does worker know which matching services are the owners of the task queues?

Temporal uses consistent hashing to assign task queue partitions to matching hosts. When the number of matching hosts changes, some partitions are reallocated according to the hashing function.

Is it recommended to auto scale matching services?
As task partitions will rebalanced if we autoscale, is there any performance implication?

Yes, it might help if matching nodes are CPU bound and you have enough partitions configured for high traffic task queues.

3

@maxim So when FE service receives the worker poll for a task queue it forwards requests to all the matching services that contains a task queue partition?

If that is the case, if there are multiple tasks in different partitions. Which task is picked up?

4

A single poll is forwarded to a single randomly chosen partition. But as you expect many poll requests in parallel, the load is usually evenly distributed. For situations when the number of pollers is very low (like 1) partitions can forward tasks and polls between themselves.

5

Thanks @maxim , that clarifies my doubt
One more question, why do we have 2 parameters
matching.numTaskqueueReadPartitions
matching.numTaskqueue WritePartitions
How is this managed under the hood at task queues level?

6

This is used to support reducing the number of partitions.
For example, a task queue has 10 partitions. Both numTaskqueueReadPartitions and numTaskqueueWritePartitions are set to 10. To reduce you change numTaskqueueWritePartitions to a smaller number. Let it run for some time, ensure that there is no backlog and then set number of read partitions to the same number as write partitions. This way no tasks are stuck in the partitions to be removed.

1 Like