I would like one Worker to pull from multiple task queues, while observing max_concurrent_activities across all queues. Is this possible using Temporal today (preferably using the Python SDK)?
The specific context behind the request is that I would like to implement some basic version of “fairness” using Temporal queues. Multiple groups (users / tenants) can generate tasks, and we would like for some users to be able “cut” a long line which was created by other users. The most basic way to achieve this is to use multiple queues, and allow only certain groups to utilize certain queues. I did see this GitHub issue for priority queues, which would be great. But in the mean time I’m looking for a stop-gap solution.
One worker pulling from multiple queues in a round-robin (or even random) fashion would be sufficient to solve this problem for us for now. From the Python SDK Workercode, it looks like this is not possible, but I thought I would double check before creating a GitHub issue.
Two follow-up questions, whenever you have the time:
Do you have an ETA on the support for fairness, and a write-up of what the fairness feature will look like?
Although it’s not implemented today, would it be doable to have the server reply to the long poll request with the events from more than 1 queue? That is, have the server implement round-robin queue polling in response to a multi-queue request from the Worker client? I ask because I would be happy to help contribute that feature, if that feature would be useful and you’d estimate the difficulty to not be too large.
I believe it should happen this year, but no hard guarantee.
Ability to subscribe to multiple queues was discussed, but it is not trivial to implement and we decided to focus on the direct support for priorities and fairness in the matching engine. The round-robin polling will not work as each queue lives in its own partition, and a naive implementation will cause contact race conditions.