Should there be any concern for “poison messages”? I assume task queues do not operate in a purely sequential manner and therefore this is not a concern?
If this is the case though, how are tasks normally processed? Is it sequentially through the backlog for a given task queue except in the case of retries?
Task queues do not provide any guarantee of the ordering of messages. The poison pills are not an issue as a failed task is retried not by a queue itself but by the workflow mutable state that reenqueues it back into the same task queue.
Should there be a concern around task queue saturation then? I assume so based on the Github issue around task queue priorities - work is generally assigned in the order they entered in the task queue, and not by any even splitting by workflow.
So if a workflow created 1k activities for a task queue and a later workflow created 10 activities, workers would probably have to process those 1k before moving to any other one?
For performance reason the newly added tasks are matched to the poll requests first and only after that tasks from the backlog are matched. So it is possible to get into situation when backlog is never drained if workers are under provisioned.
How do we determine if a task is new or on backlog?
Do retries get matched to poll request first as well?
How do we determine if a task is new or on backlog?
An activity attempt is part of its info structure. In the Go SDK it is accessed through activity.GetInfo().Attempt, in the Java one through ActivityExecutionContext.getInfo().getAttempt().
Do retries get matched to poll request first as well?
No, retries are not different from any other activity tasks from the task queue point of view.