Queue Semantics

Should there be any concern for “poison messages”? I assume task queues do not operate in a purely sequential manner and therefore this is not a concern?

If this is the case though, how are tasks normally processed? Is it sequentially through the backlog for a given task queue except in the case of retries?

Task queues do not provide any guarantee of the ordering of messages. The poison pills are not an issue as a failed task is retried not by a queue itself but by the workflow mutable state that reenqueues it back into the same task queue.

Should there be a concern around task queue saturation then? I assume so based on the Github issue around task queue priorities - work is generally assigned in the order they entered in the task queue, and not by any even splitting by workflow.

So if a workflow created 1k activities for a task queue and a later workflow created 10 activities, workers would probably have to process those 1k before moving to any other one?

For performance reason the newly added tasks are matched to the poll requests first and only after that tasks from the backlog are matched. So it is possible to get into situation when backlog is never drained if workers are under provisioned.