In what situation should we use multiple separated task queues?

Support
#1

We have one Java repo with a few workflows and activities. We have one worker class with main method that registers all workflows and activities on one queue. We build a fat jar and put it in a docker image and deploy it onto our K8s cluster. It’s working OK so far. To have separated queues requires us to have separate worker class, build different jars, build different docker images, have separated deployments. A lot of work.

So, my question is in what case should we consider have multiple task queues? Is there any problem we will run into with what we are doing now?

#2

In your case using a single task queue for all activities and workflows sounds like the right choice.

Some reasons to use more than one task queue

  • Multiple deployment units (services). If you want to deploy some set of activities and workflow types in a separate pool of processes they have to listen on a separate task queue. It can be done due to organizational boundaries or due to performance reasons. For example, some activities might require expensive GPU resources and running them separately from others is cheaper.

  • Rate limiting and flow control. If some set of activities needs to be rate limited and flow controlled independently from others. For example, an activity can be calling some downstream service that imposes a maximum allowed rate limit. As rate limiting is per worker/task queue this activity has to be using its own worker and task queue name.

  • Routing to specific hosts. See the file processing sample which uses host specific task queues for activity affinity.

1 Like