# Clarification: combination max\_concurrent\_activities and start\_to\_close\_timeout

**URL:** <https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274>\
**Category:** Community Support\
**Created:** [August 24, 2024, 4:26pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274 "2024-08-24T16:26:33Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 24, 2024, 4:26pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/1 "2024-08-24T16:26:33Z")

</div>

Hi Temporal guys!

I don’t understand well the combination of the two configuration  
`max_concurrent_activities`/`max_concurrent_workflow_tasks` and `start_to_close_timeout`.

In my scenario, I need to limit each worker instance to process only a set number of activities, based on my `settings.TEMPORAL_NUM_WORKERS`, because my machines have limited resources and the operations are CPU-intensive. Workflows and activities are currently 1-to-1.

The problem is for example that when a client with one worker instance tries to start 8 workflows and `settings.TEMPORAL_NUM_WORKERS` is set to 2, all 8 workflows start their timeout countdown immediately. I want only 2 workflows to run at a time, and the timeout for the others should start only after a previous one finishes.

Since the workflows can take anywhere from 0 to 5 minutes, if the timeout countdown starts before they’re actually processed, the other activities end up accumulating the execution time of the previous ones.

my code is this:

```python
@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self, data: WorkflowData) -> str:
            result = await workflow.execute_activity(
                my_cpu_intensive_activity,
                data,
                start_to_close_timeout=timedelta(minutes=5),
                task_queue=settings.TEMPORAL_QUEUE,
                retry_policy=RetryPolicy(maximum_attempts=1),
            )
            return result

```

```python
        worker = Worker(
            sdk_client,
            task_queue=settings.TEMPORAL_QUEUE,
            activities=[my_cpu_intensive_activity],
            workflows=[MyWorkflow],

            activity_executor=ProcessPoolExecutor(max_workers=settings.TEMPORAL_NUM_WORKERS),

            max_concurrent_workflow_tasks=settings.TEMPORAL_NUM_WORKERS,
            max_concurrent_workflow_task_polls=settings.TEMPORAL_NUM_WORKERS,
            max_concurrent_activities=settings.TEMPORAL_NUM_WORKERS,
            max_concurrent_activity_task_polls=settings.TEMPORAL_NUM_WORKERS,
            
            shared_state_manager=SharedStateManager.create_from_multiprocessing(
                multiprocessing.Manager()
            ),
        )
        await worker.run()

```

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/1/1cd6e1658a79240799c24b99ed5127a99dabb0d8.png)  
Based on the image above, my mental model was:  
if I set `max_concurrent_activities` and `max_concurrent_workflow_tasks` to `2` and have 8 potential workflows (started with `temporal_client.start_workflow(...)`).  
I expected only 2 to start, with the remaining 6 being scheduled but waiting for the current ones to finish before starting.  
Am I misunderstanding how this works, or is there something wrong with my configuration?

```auto
expected:
-------------------------------------------------------------------------------------------------------
STARTED | STARTED | SCHEDULED | SCHEDULED | SCHEDULED | SCHEDULED | SCHEDULED | SCHEDULED |  
-------------------------------------------------------------------------------------------------------
current:
-------------------------------------------------------------------------------------------------------
STARTED | STARTED | STARTED | STARTED | STARTED | STARTED | STARTED | STARTED |  
-------------------------------------------------------------------------------------------------------

```

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 26, 2024, 6:24am UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/2 "2024-08-26T06:24:51Z")

</div>

The confusion comes from misunderstanding what the workflow task is. Each workflow is executed as a sequence of workflow tasks that are created when there are new events to process. Limiting the number of parallel workflow tasks _does not_ limit the number of parallel workflows. It limits the throughput and resource utilization (only for workflow code) per worker. We recommend not changing the default value of `max_concurrent_workflow_tasks` for use cases like yours. Changing the number of pollers is also not needed unless you need to run a very high throughput scenario.

Currently, Temporal doesn’t directly support limiting the number of parallel workflow instances. One possible workaround is to create a workflow that starts the processing workflows in a controlled manner. See [batch samples](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch) from the java-samples repository.

---

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 26, 2024, 2:29pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/3 "2024-08-26T14:29:21Z")

</div>

hi @maxim,

Thanks so much for your answer! Now it’s more clear.

now, some points:

### Batch samples

I couldn’t understand it fully. I tried to figure it out by looking at the [batch samples](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/batch), but from what I understand, the samples seem to process elements within the same workflow execution “message”. I might be wrong(I need to overcome my learning curve with Temporal), but it in my scenario, I can’t group ‘my items to be processed’ into pairs or triples… from the client, as I don’t have control over them. Let me know if my intuition is off.

### A possible solution?

I found a solution, but it requires a global worker semaphore, something like:

```python
with workflow.unsafe.imports_passed_through():
    from temporal_project.multiprocessing_worker_semaphore import WorkerSemaphore

@workflow.defn
class MyWorkflow:
    @workflow.run
    async def run(self, data: WorkflowData) -> str:
        await WorkerSemaphore.acquire()
        try:
            result = await workflow.execute_activity(
                my_cpu_intensive_activity,
                data,
                start_to_close_timeout=timedelta(minutes=5),
                task_queue=settings.TEMPORAL_QUEUE,
                retry_policy=RetryPolicy(maximum_attempts=1),
            )
         except Exception as error:
            WorkerSemaphore.release()
            raise error

        WorkerSemaphore.release()
        return result

```

To implement this, I wanted to use Python’s `Semaphore` from `asyncio`, but since each workflow has its own event loop, an event loop-level lock isn’t suitable. So, I used a `multiprocessing` lock, hoping it won’t interfere with the underlying workings of the Temporal worker.

I don’t know if its an optimal solution:

```python
class WorkerSemaphore:
    free_workflows_number: multiprocessing.Value = None
    lock: multiprocessing.Lock = None

    @staticmethod
    def set_max_parallel_workflows_number(manager: multiprocessing.Manager, max_parallel_workflow_number: int):
        WorkerSemaphore.lock = manager.Lock()
        WorkerSemaphore.free_workflows_number = manager.Value("i", max_parallel_workflow_number)

    @staticmethod
    async def acquire():
        while True:
            with WorkerSemaphore.lock:
                if WorkerSemaphore.free_workflows_number.value > 0:
                    WorkerSemaphore.free_workflows_number.value -= 1
                    return
            await asyncio.sleep(20)

    @staticmethod
    def release():
        with WorkerSemaphore.lock:
            WorkerSemaphore.free_workflows_number.value += 1

```

In my tests, it appears to work well, but I’d like feedback from experts. Using a semaphore for this seems a bit wrong to me… Also, the web panel indicates that workflows are polling every tick seconds to be ready for processing activities. While this is useful, I’m worried I might be falling into an anti-pattern. What do you think?

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/5/50a416dce7e64fd82d962eb2f1be17995041d2ab.png)

### Worker configuration

so, to better understand the worker configuration in my usecase:

```python
activity_executor=ProcessPoolExecutor(max_workers=settings.TEMPORAL_NUM_WORKERS),

max_concurrent_workflow_tasks=settings.TEMPORAL_NUM_WORKERS, max_concurrent_workflow_task_polls=settings.TEMPORAL_NUM_WORKERS,
max_concurrent_activities=settings.TEMPORAL_NUM_WORKERS,
max_concurrent_activity_task_polls=settings.TEMPORAL_NUM_WORKERS,

```

Given that my activity involve thread blocking, thread spawning, and is CPU-intensive, and since I don’t control its internals (as it come from a third-party library and I can’t implement a heartbeat in the middle of its execution), I need to configure the `activity_executor` accordingly. I plan to adjust the `TEMPORAL_NUM_WORKERS` setting to a ratio of 1:2 relative to the number of cores on the worker machine.

So, the only configuration I’m unsure about is `max_concurrent_activities`. Although it controls the number of concurrent activities within a single workflow, the app issues a warning if this number is bigger than the `ProcessPoolExecutor(max_workers=)`…

So, is this configuration okay?

```python
activity_executor=ProcessPoolExecutor(max_workers=settings.TEMPORAL_NUM_WORKERS),
max_concurrent_activities=settings.TEMPORAL_NUM_WORKERS,

```

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 26, 2024, 4:19pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/4 "2024-08-26T16:19:57Z")

</div>

You cannot use any global variables inside the workflow code. Also, note that workflows are distributed across multiple hosts, so such a semaphore cannot work by definition.

What is the maximum rate (per second) and number of parallel workflows that you need to support for this use case?

---

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 26, 2024, 5:00pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/5 "2024-08-26T17:00:12Z")

</div>

It’s fine for me to limit the number per worker instance. The idea is that if I have 4 worker instances and each can process 2 workflows at a time, then all workers combined can handle 8 workflows simultaneously.

It’s difficult to determine the exact rate per second because, in my scenario, users have a cap on the maximum number of jobs they can run in parallel, with each job lasting anywhere from 0 to 5 minutes.

So, if I have 5 users with a maximum of 10 background operations each, they will likely want to use their full capacity. I could receive 50 requests initially, with new requests arriving as soon as previous jobs finish.

The issue is that the CPU-intensive activity slows down or crashes the application if more than 2 activities are run in parallel on a machine with 4 cores, for example.

But in practice is that the timeout countdown for each task starts immediately, instead of waiting for the computation resources to process the next activity to become available.

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 26, 2024, 5:25pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/6 "2024-08-26T17:25:03Z")

</div>

> But in practice is that the timeout countdown for each task starts immediately, instead of waiting for the computation resources to process the next activity to become available.

If you limit the number of activities per worker then this cannot happen. The activity StartToClose timeout starts only when there is a free slot in that worker.

> It’s fine for me to limit the number per worker instance. The idea is that if I have 4 worker instances and each can process 2 workflows at a time, then all workers combined can handle 8 workflows simultaneously.

There is no way to limit number of workflows per worker instance. You have to do it yourself. If the rate is low you can create a special “mutex” workflow to act as semaphore.

---

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 26, 2024, 5:33pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/7 "2024-08-26T17:33:37Z")

</div>

1. If I understand correctly, the activity limit applies within a single workflow. However, since each of my activities corresponds to one workflow, 8 workflows mean 8 activities. The configuration doesn’t seem to affect activities across different workflows.

2. Do you have any guidance or documentation on creating a semaphore/mutex workflow without using a global variable?

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 26, 2024, 6:02pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/8 "2024-08-26T18:02:30Z")

</div>

1. Incorrect. The worker configuration applies to activities that are executed by a worker and is not related to workflows in any way.
2. Here is a [Go sample](https://github.com/temporalio/samples-go/tree/main/mutex).

---

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 26, 2024, 6:13pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/9 "2024-08-26T18:13:33Z")

</div>

> [@maxim](#):
>
> Incorrect. The worker configuration applies to activities that are executed by a worker and is not related to workflows in any way.

If that’s true, I wouldn’t need a mutex :). However, in my current worker, I just started 4 workflows, and all 4 activities started instantly, even though `max_concurrent_activities` is set to 2. Could I have misconfigured the worker about synchronous tasks or something else that might be causing this effect?

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 26, 2024, 6:19pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/10 "2024-08-26T18:19:49Z")

</div>

Please check the identity field at ActivityTaskStartedEvent for these four activities. I suspect you have more than one worker running.

---

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 26, 2024, 6:46pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/11 "2024-08-26T18:46:05Z")

</div>

![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/0/01798c4328796d441f35aa38c4ed690131dc047c.png)

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/e/ed82ac7d5d06418ae83613931acbc686945f6e76.png)  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/d/d51d170a2dddc153d61c288319eff8b76aab857c.png)  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/1/18ac4fcd3342a82d38cba5ffa8077922a00f85b4.png)  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/7/7c6ac7523fbb7d3f2936784956658fa2379e39c0.png)

No, the worker is one, but I notice that the start timestamp is accurate 🤔.

My issue is with the timeout for pending activities:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/1/12eb42b255ccbc8af8b1c9b4811374d9d5729320.png)

Could it not be related to `start_to_close_timeout`? I just modified the `start_to_close_timeout`, but this expiration remains unchanged.

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 27, 2024, 4:34am UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/12 "2024-08-27T04:34:34Z")

</div>

> [@AngeloChecked](#):
>
> ```auto
> start_to_close_timeout=timedelta(minutes=5),
> retry_policy=RetryPolicy(maximum_attempts=1),
> 
> ```

The above means not to retry the activity in case of failure and wait 5 minutes before reporting the timeout. I’m not sure what the question is.

---

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 27, 2024, 7:47am UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/13 "2024-08-27T07:47:58Z")

</div>

hi @maxim,

Nothing. Thanks for all your support; you’ve been super helpful. The problem was with the `run_timeout` in the client code, which overrides the activity code’s timeouts ( This configuration was setting both `ScheduleToCloseTimeout` and `ScheduleToStartTimeout` to 5). I guess I got a bit too creative with my leaps of logic 😅.

```auto
        await self.temporal_client.start_workflow(
            "MyWorkflow",
            JobEvent(body, job_id),
            id=job_id.workflow_id,
            task_queue=self.settings.TEMPORAL_QUEUE,
            run_timeout=timedelta(minutes=5),
        )

```

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 27, 2024, 9:06am UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/14 "2024-08-27T09:06:10Z")

</div>

We recommend not setting a workflow timeout at all. Would you really want your workflow terminated without any ability to cleanup if your worker was down for 5 minutes?

---

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 27, 2024, 9:33am UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/15 "2024-08-27T09:33:18Z")

</div>

I don’t have control over the user input for background operations. While they should generally take less than 5 minutes, complex inputs could cause activities to run for hours. Therefore, I need to enforce an execution timeout to ensure the computation time stays within limits. Is there a better solution?

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 27, 2024, 11:04am UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/16 "2024-08-27T11:04:15Z")

</div>

Who implements the activities? Can they fail if the input is not the one you want to support?

---

<div class="post-metadata">

**Author:** ![AngeloChecked](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/angelochecked/32/5707_2.png) [@AngeloChecked](https://community.temporal.io/u/AngeloChecked)\
**Post date:** [August 27, 2024, 12:44pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/17 "2024-08-27T12:44:14Z")

</div>

The input is user-defined, similar to HTML, with a lot of variety, and I don’t currently have effective heuristics to determine if the job will fail or time out before processing it.

---

<div class="post-metadata">

**Author:** ![maxim](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/maxim/32/8_2.png) [@maxim](https://community.temporal.io/u/maxim)\
**Post date:** [August 27, 2024, 1:18pm UTC](https://community.temporal.io/t/clarification-combination-max-concurrent-activities-and-start-to-close-timeout/13274/18 "2024-08-27T13:18:59Z")

</div>

Use activity heartbeating. This way, you can have activity that can run for a long time, but timeout if the process crashes or hangs.