Limit Concurrent Activities in Parallel

In Golang, there’s the popular concept of worker pools: Go by Example: Worker Pools. Is there a way to implement this in Temporal?

Let’s say that you have a batch of activities to execute but only want 4 activities executing at the same time, is this possible?

1 Like

Set WorkerOptions.MaxConcurrentActivityExecutionSize to limit number of activities executing at the same time per worker.

Hmm - this only allows us to limit activities per worker, but what if you wanted to limit the activities on a workflow?

For example, if you have a workflow that needs to execute 64 activities, but only want 4 activities executing concurrently. Is that possible?

Also, is there a limit on how many activities you can have running in parallel at any point in time?

Actually, is that what the “swarm” example here is: samples-go/swarm.go at e89517ba16b481a03b5a0b4edb8733aab6b135e3 · temporalio/samples-go · GitHub?

I haven’t heard of swarm before, so maybe that terminology is why I didn’t jump to this. Do you have a rough description of what this attempts to do?

Hmm - this only allows us to limit activities per worker, but what if you wanted to limit the activities on a workflow?
For example, if you have a workflow that needs to execute 64 activities, but only want 4 activities executing concurrently. Is that possible?

You start 4 activities and start more when one of the executing activities is completed. Note that in Go SDK ExecuteActivity returns a Future. So the call is already asynchronous. You can use Selector for waiting on multiple Futures.

Also, is there a limit on how many activities you can have running in parallel at any point in time?

Limiting the number of executing activities across multiple workers is not supported. Only a global rate list per task queue is supported.

I haven’t heard of swarm before, so maybe that terminology is why I didn’t jump to this. Do you have a rough description of what this attempts to do?

It was contributed. So I’m not an expert in the mathematical model it implements.

I see - so you could start 4 activities, then with the Selector, you can just kick off another one as an activity finishes. This seems to be a good fit for our case, but is there a reason you wouldn’t call this a cap on the activity parallelism? Is it because this isn’t a global activity limit (i.e. across all workflows) but only a limit on the parallelism of the single workflow?

This seems to be a good fit for our case, but is there a reason you wouldn’t call this a cap on the activity parallelism? Is it because this isn’t a global activity limit (i.e. across all workflows) but only a limit on the parallelism of the single workflow?

Yes, I was talking about the global cap on activity parallelism which applies to multiple workflows.

On this topic, in Java, is this a reasonable way to pause the workflow if too many activities are queued up, or is it a busy-wait?

Workflow.await(() -> activities.stream().filter(p -> !p.isCompleted()).count() < MAX_RUNNING)

This is in a cpu-intensive workflow, where WorkerOptions.MaxConcurrentActivityExecutionSize is set to the number of cpu cores. The list of activities far exceeds that and I’d rather not have the workflow get too far ahead of the queue.

That await should work fine. It is not a busy-wait as the condition is evaluated only if the state of the workflow is changed due to some external event like activity completion.