Is the thread pool shared among all Workers created from the same WorkerFactory?

Community Support
1

Here’s some context to help understand why I’m asking this:

We want to start with a simple deployment where all workers are deployed in the same JVM, and then scale-out as needed. This means that I want to have all workflows and activities specified in a way that I can start dividing them into different workers as needed, at deployment time, without the need to change workflow implementation code. This would also allow to configure different deployments as needed (for example, using a generic Worker launch implementation that receives the task list name, workflow class and activity class and other settings by configuration/parameters).

One way of doing it dynamically would be to use a single task list name. However this cannot be done because then each Worker would have to register all workflows and activities implementation for every type of task, which destroys the initial goal
So the only way to allow such separation is by having independent task list names since the very beginning.
In this case, to start with a single JVM containing all workflow and activities workers, I was planning to start a single WorkerFactory, and then create a Worker per task list name.

But my question is whether I can optimize the hardware resources usage with this approach.
If there’s a single thread pool in the WorkerFactory that is shared with all Workers created from that factory, then I’m sure the thread pool will be optimized. But if each Worker has its own thread pool then I would need to reduce the thread pool of each worker to avoid overleading the JVM and then I may ending up having most of the threads from other thread pools in idle state.

I would bet that the thread pool is per WorkerFactory, but in the API I can only specify thread pool settings per Worker so I’m not sure how is actually implemented.

Note: if you have a better suggestion in how to implement this use-case please let me know.

Thank you in advance!

3

Your requirements of not requiring workflow updates when activity deployment topology changes are spot on. We have plans to support a level of indirection that would allow changing activity task queue name without through a proposed metadata service. So the workflow is going to specify an activity type only and then the mapping will happen on the service.

Unfortunately, it is not currently implemented yet and the best option is to use some configuration to make a queue name that a workflow uses configurable.

It is not feasible to use a single thread pool to execute activities from multiple task queues. The reason is that a poller doesn’t even emit a poll request if its thread pool is full. Doing it for multiple pollers that use a long poll on top of a single thread pool is not really possible.

4

Ok, got it.
We can go with a configuration approach, that should work.
However, we were planning to start with Cadence now using some abstractions and move later in a few months to Temporal since we don’t have a ETA for Temporal yet. But in Cadence the task list name is defined in annotations so this means to change the task list names we need to restart the processes, which is acceptable for this scenario.
Thanks.

5

You don’t have to use annotation to specify task list in Cadence as well. It can be specified through the ActivityOptions.

6

I thought this was only possible in Temporal API.
Good to know.
Thanks!