Worker processes with multiple worker entities polling on different queues

Community Support
1

I need a worker to listen to two queues so that I can throttle them. There were no examples of how I can run two worker entities in a single worker process. On the worker docs page it says:

A single Worker Entity can listen to only a single Task Queue. But if a Worker Process has multiple Worker Entities, the Worker Process could be listening to multiple Task Queues.

This is how I have set it up now, but not sure if that’s the right way to do it.

func main() {
     …
     …
     err = w.Start()
     err2 := w2.Start()
     if err != nil {
         logger.Fatal(“Unable to start worker”, err)
     }
     if err2 != nil {
         logger.Fatal(“Unable to start worker”, err2)
     }
     <-worker.InterruptCh()
}

2

Hello @karthik

The code you have shared will start two workers within the same process and as you have mentioned you can have those workers listening on different taskqueues.

I need a worker to listen to two queues so that I can throttle them

Can you elaborate on this? sorry I don’t understand how this is related to having two workers in the same process,

Antonio

3

Sorry about this

I need a worker to listen to two queues so that I can throttle them

I meant a worker process with two worker entities listening to two queues.
Just to clarify the terminology here, the two workers I have started in the code, are they the worker entities? and the process running these two is the worker process?

One follow up question:

If I have two workers in the process with each listening on their own queues, I am assuming if I set MaxConcurrentActivityExecutionSize and TaskQueueActivitiesPerSecond for one of the workers to half of what it is set to the other worker, I will see an increase in throughput from the queue that has the higher MaxConcurrentActivityExecutionSize and TaskQueueActivitiesPerSecond?

4

yes, that is what I understand.

I would say it depends. are activities placed in both taskqueues consuming the same amount of resources, same end-to-end latency etc…

Note that they are two different things with two different proposes:

5

Thanks @antonio.perez . That is what I was looking for

6

Hi, Just wanted to see if this is achievable in ruby-sdk? I tried starting the 2 workers but until the first worker is terminated the temporal ruby-sdk doesn’t allow to start the second worker entity under the same worker process.

How can we achieve this worker process polling with worker entities on different queues in ruby-sdk?