Is it possible for a worker to have multiple task queues?

Community Support
, ,
1

Is it possible for a Worker Entity polling for Tasks to one or more Task Queues?

According to What is a Temporal Worker? | Temporal Documentation,

A Task Queue is a lightweight, dynamically allocated queue that one or more Worker Entities poll for Tasks

Say you create a parent workflow that spawns millions of child workflows, each with 3 activities. The worker.go file would look something like this using the Go SDK:

func main() {
	c, err := client.Dial(client.Options{})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}
	defer c.Close()

	w := worker.New(c, app.MyTaskQueue, worker.Options{})

	w.RegisterWorkflow(ParentWorkflow)
	w.RegisterWorkflow(ChildWorkflow)
	w.RegisterActivity(Activity1)
	w.RegisterActivity(Activity2)
	w.RegisterActivity(Activity3)

	err = w.Run(worker.InterruptCh())
	if err != nil {
		log.Fatal("Unable to start worker", err)
	}
}

So here, you have to create a worker with a specified Task Queue. Going back to my question, which was derived from my assumption that having multiple Task Queues can help execute tasks faster, is it possible for a Worker Entity to allocate Tasks to multiple task queues?

2

Going back to my question, which was derived from my assumption that having multiple task queues can help execute tasks faster, is it possible for a Worker Entity to allocate Tasks to multiple task queues?

Your assumption is not correct. A single task queue can scale as high as needed.

3

Thank you very much for answering.

The revised question is can I have a workflow which spans activities that are bound to different task queues?

On my analysis of task queues, there are two things that I need to consider, the first is prioritization, especially since temporal task queues have no guarantee of ordering and the second is whether we will have all our workers be the same, or if we will dedicate some to specific tasks especially since worker entities listening to the same task queue must be registered to handle the exact same WF Types and Activity Types.

4

Yes, a workflow can specify a task queue through ActivityOptions when scheduling an activity.

5

I got a bit confused by your answer. Let me make sure to check. Is it possible for a workflow to specify multiple task queues through ActivityOptions when scheduling activities?

6

Yes, it is possible. In Go, you do it by creating a new context for each activity invocation.

7

Iā€™m clear. Thank you very much for your time.

8

Sorry again, I have one more question.

Can I have a workflow with 3 activities that are bound to 3 workers like activityA that is bound to workerA and allocated to a task queueA, activityB that is bound to workerB and allocated to a task queueB, activityC that is bound to workerC and allocated to a task queueC,

9

Yes, it will look something like:

	optionsA := workflow.ActivityOptions{
		TaskQueue: "A",
                  ...
	}
	ctxA = workflow.WithActivityOptions(ctx, optionsA)
	_ = workflow.ExecuteActivity(ctxA, A).Get(ctx, ...)

	optionsB := workflow.ActivityOptions{
		TaskQueue: "B",
                  ...
	}
	ctxB = workflow.WithActivityOptions(ctx, optionsB)
	_ = workflow.ExecuteActivity(ctxB, B).Get(ctx, ...)

	optionsC := workflow.ActivityOptions{
		TaskQueue: "C",
                  ...
	}
	ctxC = workflow.WithActivityOptions(ctx, optionsC)
	_ = workflow.ExecuteActivity(ctxC, C).Get(ctx, ...)
1 Like