How to limit instances of workers

Community Support
1

Hello,

Could you please point me in the direction of how to configure my worker instances. I think I have a very vanilla worker but at one point during its execution it intentionally uses 2Gb. I would like to limit the number of simultaneous worker instances.

At the moment if I start one worker and submit runs, both runs happen together. If enough users run something and there are not enough workers, out of memory.

Sorry if this is a very basic question, I am new and could not find the answer so far.

Matt

2

This might be it.

WorkerFactoryOptions options = WorkerFactoryOptions.newBuilder()
		    										.setWorkflowCacheSize(max)
		    										.build();
WorkerFactory factory = WorkerFactory.newInstance(client, options);

However each workflow has several activities and only one is the large memory one. I could put a semaphore in its code but I feel like their should be a way to limit it…

3

You cannot specify specific activities limits. All limits are per worker. Use WorkerOptions.MaxConcurrentActivityExecutionSize to limit number of parallel activities per worker.

If you have just one activity that requires a limit create a separate worker just for it. Make sure that this worker uses a separate task queue name. When invoking this activity from a workflow specify this task queue through ActivityOptions.

1 Like
5

Yes it is just one activity from several. Perhaps there is an example of calling one workflow from another in the examples? This would help me but I didn’t see it (examples are great through:). A static Semaphore in the offending activity of size 1 does also work as far as I can tell.

6

Perhaps there is an example of calling one workflow from another in the examples?

Yes, here is a child workflow example.

A static Semaphore in the offending activity of size 1 does also work as far as I can tell.

When using MaxConcurrentActivityExecutionSize activity is not started until a worker has spare capacity. When a semaphore is used the activity is started and its timer is ticking while it is blocked on the semaphore. So I would recommend avoiding the semaphore approach.

1 Like
7

I spent time on this this morning. Just to let you know it took maybe an hour to change my code around and appears to pass the memory tests I have fine. Thanks very much.

1 Like