Hi team,
Is there any specific order in which the workers pick scheduled task from the task queue ?
I was curious, how will temporal workers react if we schedule huge number of activities and workflow tasks all of a sudden for a workflow? Will it lead to starvation of task scheduled by other workflows in the same task queue ?
For workflows, currently, workflow task order does not guarantee workflow execution order (because tasks can fail and get retried etc which can break this order).
If you need strict ordering of workflow executions typical recommendation is to do that inside workflow code (where you can control exact child workflow execution ordering).
For activities, Temporal does not provide guarantees of activity execution ordering. For example you cannot guarantee that activity retries happen in some specific order (when you have multiple parallel activities executed by worker). To prevent overloading downstream services, Temporal does provide task queue rate limiting.
If you want to rate limit a specific activity have it listen on its own task queue and set WorkerOptions->TaskQueueActivitiesPerSecond (default 100K in Go SDK) to N to get N activities per second rate limit.
Note that there are features that will come in which will allow you to control this better, see for example here and here.
To add, Temporal workers do scale horizontally quite well and can handle a large number of concurrent workflow executions. See worker tuning guide in docs for more info on what things to look out for when in need to scale your workers.
Thanks for you detailed insights. My concern of ordering was mainly from an isolation angle, so that a high through-put workflow doesn’t lead to low throughput workflow starvation. Which I assume might happen. Correct me if I am wrong ?
I think this issue correctly describe the problem, for which I have been seeking the answer for. Thanks for sharing.
The linked issue is correct. Temporal doesn’t provide any hard fairness guarantees or prioritization across multiple workflows at this point. We might add such features, but nothing is planned.
Thanks for the confirmation.