Difference Between limiting Max Concurrent Activity Execution Size and limiting number of Workers?

Community Support
1

We are looking to limit the total number of active activities, for a certain type of activity.

We are running a single Go process to pick and run up workflow and task executions.
We are going to have:

  • one task queue for all the little setup and cleanup activities that are short-lived
    • we can tolerate many of these being done concurrently
    • we don’t want these tasks waiting to be done because our workers are tied up with the long-running tasks - once a long-running activity completes, its corresponding cleanup activities should be a fast follow.
  • a second task queue for the type of activity that we want to rate limit: essentially start (long running intensive process) and wait for completion (of long running intensive process)
    • the start activity is very short, it dispatches the command to an external process and returns, but we group it in this queue to avoid starting any new activities while our desired max number of this type of activity is running
    • the wait for completion activity sits on a stream until a certain status (completed or failed) is written to the stream by an external process

For a simple example, in the Go runtime, Is there an effective difference between registering 20 workers for the intensive activity queue, each with a MaxConcurrentActivityExecutionSize of 1, and say 1 worker with MaxConcurrentActivityExecutionSize of 20? or in between, like 4 workers with MaxConcurrentActivityExecutionSize 5?

Thanks

2

Given that the activities are not CPU/memory intensive they can fit a single process without problem. So you can have a single worker with max execution size of 20. Having 20 workers with max size of 1 will achieve the same result, but would be more expensive for you.

1 Like
3

Thank you - I had assumed that the workers in the Go runtime were just goroutines, so those should still be pretty inexpensive overall, right?

4

Workers are pretty complex objects. Each activity invocation is done in its own goroutine.

1 Like