How to limit instances of workers

Hello,

Could you please point me in the direction of how to configure my worker instances. I think I have a very vanilla worker but at one point during its execution it intentionally uses 2Gb. I would like to limit the number of simultaneous worker instances.

At the moment if I start one worker and submit runs, both runs happen together. If enough users run something and there are not enough workers, out of memory.

Sorry if this is a very basic question, I am new and could not find the answer so far.

Matt

This might be it.

WorkerFactoryOptions options = WorkerFactoryOptions.newBuilder()
		    										.setWorkflowCacheSize(max)
		    										.build();
WorkerFactory factory = WorkerFactory.newInstance(client, options);

However each workflow has several activities and only one is the large memory one. I could put a semaphore in its code but I feel like their should be a way to limit it…

You cannot specify specific activities limits. All limits are per worker. Use WorkerOptions.MaxConcurrentActivityExecutionSize to limit number of parallel activities per worker.

If you have just one activity that requires a limit create a separate worker just for it. Make sure that this worker uses a separate task queue name. When invoking this activity from a workflow specify this task queue through ActivityOptions.

1 Like

Yes it is just one activity from several. Perhaps there is an example of calling one workflow from another in the examples? This would help me but I didn’t see it (examples are great through:). A static Semaphore in the offending activity of size 1 does also work as far as I can tell.

Perhaps there is an example of calling one workflow from another in the examples?

Yes, here is a child workflow example.

A static Semaphore in the offending activity of size 1 does also work as far as I can tell.

When using MaxConcurrentActivityExecutionSize activity is not started until a worker has spare capacity. When a semaphore is used the activity is started and its timer is ticking while it is blocked on the semaphore. So I would recommend avoiding the semaphore approach.

1 Like

I spent time on this this morning. Just to let you know it took maybe an hour to change my code around and appears to pass the memory tests I have fine. Thanks very much.

1 Like