Creating multiple workers in the same process

Community Support
1

I’m thinking of creating multiple worker processes using a for loop to listen and poll from the same task-queue. However, all the workers created have a default ID that is based on the client identity/pod name. Based on the logs, multiple workers were started but upon checking the temporal web UI, it only shows 1 active worker.

Does this create multiple worker entities correctly and is this the right practice?


for i := 0; i < numWorkers; i++ {
		w := worker.New(client, task-queue, workerOptions{})
		err := w.Start()
		if err != nil {
			return err
		}
	}
2

Why are you creating multiple workers for the same task queue in the same process? This is not ever needed.

3

Would having multiple workers for the same task queue in the same process allow the workers to process the more tasks from the same task queue?

4

No, it doesn’t bring any advantage.

5

Why though?

6

Because a single worker can saturate the process CPU/memory.

7

I have a related question. We have a requirement to execute activities (from different workflows) in parallel. In this case, is the recommendation to create separate task queues for each workflow and have one worker per task queue? (Instead of having a single task queue for all workflows, and multiple workers read from the same task queue)

8

I would start from a single task queue for everything and move to multiple task queues only if you want to break your processes into multiple binaries (services) or have specific rate-limiting requirements for a subset of activities.

9

If I have a single task queue for all workflows, how do I achieve parallel execution of activities with a single worker? By registering the worker for all activities?

10

Yes, you register all the activities and workflows with the same worker.

1 Like
11

Understood! Thank you for your quick response.