When the task volume is greater than the number of concurrent activities, the subsequent activity execution time does not reach the start_to_close_timeout and is directly canceled

Community Support
,
1

My activity tasks are concurrent with 10. If the number of ExecuteWorkflow declarations is greater than 10 at the same time, for example 20, my first 10 activities will be executed normally, and the next 10 activities will be Cancelled before the start_to_close_timeout is reached.

It looks like self._bridge_worker().poll_activity_task() in temporalio/worker/_activity.py gets the activity task, and after getting it, the start timer starts. Since the thread pool is full, the subsequent tasks will be blocked, but they will continue to time, causing the subsequent tasks to be canceled before reaching the start_to_close_time.



The following is my approximate configuration：
    // golang ExecuteWorkflow
	workflowOptions := client.StartWorkflowOptions{
		ID:        req.TaskId,
		TaskQueue: Queue, // 队列名
	}
	Task := models.Task{
		TaskID:      req.TaskId,
	}

	_, err := TemporalClient.ExecuteWorkflow(
		context.Background(), workflowOptions, task.Workflow, Task)
	
	
	// python Worker and execute_activity
	with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        worker = Worker(
            temporal_client.client,
            task_queue=TASK_QUEUE,
            workflows=[TaskClass],
            activity_executor=executor,
            activities=[task_activity],
            workflow_runner=custom_runner,
        )
        await worker.run()
		
		
	@workflow.defn(name="workflow-test")
    class TaskClass:
		@workflow.run
		async def run(self, task: Task):
			task_retry_policy = RetryPolicy(
				initial_interval=timedelta(
					seconds=TEMPORAL_CONFIG.task_initial_interval // 15s
				),
				maximum_attempts=TEMPORAL_CONFIG.task_maximum_attempts, // 2
			)
			await workflow.execute_activity(
				activity=task_activity,
				arg=task,
				start_to_close_timeout=timedelta(
					seconds=TEMPORAL_CONFIG.task_execute_time # 120s 
				),
				retry_policy=task_retry_policy,
			)
2

Hi ,

could you share the workflow history with one of the cancelled activities? or do you mean the activity fails after 2 retries?

3

What I mean is that when 20 tasks arrive at the same time, max_workers is set to 10, for example, the execution time of each task is 80s, and start_to_close_timeout is set to 120s, the problem is: the first 10 tasks are completed after 80s, and the next 10 tasks are only executed for about 40s and are set to CancelledError

4 
2025-10-09 12:41:14,340.340 12112 INFO ../task_activity.py:19:task_activity() start task_activity: 9135dc0b-fb8d-42a2-a81b-9007361170aa
2025-10-09 12:41:58,395.395 12112 ERROR ../my_task_activity.py:61:handle_cancelled_error() task 9135dc0b-fb8d-42a2-a81b-9007361170aa execute exceeds 120.0 s
[trace-0a46e1632c9f41ef]Traceback (most recent call last):
[trace-0a46e1632c9f41ef][trace-0a46e1632c9f41ef]  File "../task_activity.py", line 20, in task_activity
[trace-0a46e1632c9f41ef]    task.strategy.execute_task()
[trace-0a46e1632c9f41ef][trace-0a46e1632c9f41ef]  File "../strategy.py", line 46, in execute_task
[trace-0a46e1632c9f41ef]    self.execute(task_id=self.task_id)
[trace-0a46e1632c9f41ef][trace-0a46e1632c9f41ef]  File "../strategy.py", line 145, in execute
[trace-0a46e1632c9f41ef]    self.flow_result = alarm_run(task_id)
[trace-0a46e1632c9f41ef][trace-0a46e1632c9f41ef]  File "../flow.py", line 396, in alarm_run
[trace-0a46e1632c9f41ef]    time.sleep(1)
[trace-0a46e1632c9f41ef][trace-0a46e1632c9f41ef]temporalio.exceptions.CancelledError: Cancelled



{
  "events": [
    {
      "eventId": "1",
      "eventTime": "2025-10-09T12:39:50.253818124Z",
      "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_STARTED",
      "taskId": "2928672812",
      "workflowExecutionStartedEventAttributes": {
        "workflowType": {
          "name": "workflow-test"
        },
        "taskQueue": {
          "name": "queue-test",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": {
                "task_id": "9135dc0b-fb8d-42a2-a81b-9007361170aa",
                "trigger_type": "alarm"
              }
            }
          ]
        },
        "workflowExecutionTimeout": "0s",
        "workflowRunTimeout": "0s",
        "workflowTaskTimeout": "10s",
        "originalExecutionRunId": "0c8517cd-d3f7-4b02-a607-2d613dd8269b",
        "identity": "122992@api-kt@",
        "firstExecutionRunId": "0c8517cd-d3f7-4b02-a607-2d613dd8269b",
        "attempt": 1,
        "firstWorkflowTaskBackoff": "0s",
        "header": {},
        "workflowId": "9135dc0b-fb8d-42a2-a81b-9007361170aa"
      }
    },
    {
      "eventId": "2",
      "eventTime": "2025-10-09T12:39:50.253854783Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "2928672813",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "queue-test",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
    {
      "eventId": "3",
      "eventTime": "2025-10-09T12:39:52.742335867Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "2928672854",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "2",
        "identity": "12112@tmp-kt",
        "requestId": "d153ab16-84a5-4a27-9636-b9bb7db5ebe4",
        "historySizeBytes": "403"
      }
    },
    {
      "eventId": "4",
      "eventTime": "2025-10-09T12:39:52.777696270Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "2928672860",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "2",
        "startedEventId": "3",
        "identity": "12112@tmp-kt",
        "workerVersion": {
          "buildId": "14de874560c3a2202fb30bb2b7483138"
        },
        "sdkMetadata": {
          "coreUsedFlags": [
            1,
            3,
            2
          ],
          "sdkName": "temporal-python",
          "sdkVersion": "1.14.1"
        },
        "meteringMetadata": {}
      }
    },    
    {
      "eventId": "5",
      "eventTime": "2025-10-09T12:39:52.777784956Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_SCHEDULED",
      "taskId": "2928672861",
      "activityTaskScheduledEventAttributes": {
        "activityId": "1",
        "activityType": {
          "name": "task_activity"
        },
        "taskQueue": {
          "name": "queue-test",
          "kind": "TASK_QUEUE_KIND_NORMAL"
        },
        "header": {},
        "input": {
          "payloads": [
            {
              "metadata": {
                "encoding": "anNvbi9wbGFpbg=="
              },
              "data": {
                "task_id": "9135dc0b-fb8d-42a2-a81b-9007361170aa",
                "trigger_type": "alarm"
              }
            }
          ]
        },
        "scheduleToCloseTimeout": "0s",
        "scheduleToStartTimeout": "0s",
        "startToCloseTimeout": "120s",
        "heartbeatTimeout": "0s",
        "workflowTaskCompletedEventId": "4",
        "retryPolicy": {
          "initialInterval": "15s",
          "backoffCoefficient": 2,
          "maximumInterval": "1500s",
          "maximumAttempts": 2
        },
        "useWorkflowBuildId": true
      }
    },
    {
      "eventId": "6",
      "eventTime": "2025-10-09T12:42:07.837531112Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_STARTED",
      "taskId": "2928672919",
      "activityTaskStartedEventAttributes": {
        "scheduledEventId": "5",
        "identity": "12112@tmp-kt",
        "requestId": "685edd23-7c8f-4d8c-9c22-a3d9af4a086e",
        "attempt": 2,
        "lastFailure": {
          "message": "activity StartToClose timeout",
          "source": "Server",
          "timeoutFailureInfo": {
            "timeoutType": "TIMEOUT_TYPE_START_TO_CLOSE"
          }
        }
      }
    },
    {
      "eventId": "7",
      "eventTime": "2025-10-09T12:42:07.917188550Z",
      "eventType": "EVENT_TYPE_ACTIVITY_TASK_COMPLETED",
      "taskId": "2928672920",
      "activityTaskCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "YmluYXJ5L251bGw="
              },
              "data": null
            }
          ]
        },
        "scheduledEventId": "5",
        "startedEventId": "6",
        "identity": "12112@tmp-kt"
      }
    },
    {
      "eventId": "8",
      "eventTime": "2025-10-09T12:42:07.917199061Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_SCHEDULED",
      "taskId": "2928672921",
      "workflowTaskScheduledEventAttributes": {
        "taskQueue": {
          "name": "12112@tmp-kt-12e959e874744652ae08109ffb2fb9bc",
          "kind": "TASK_QUEUE_KIND_STICKY",
          "normalName": "queue-test"
        },
        "startToCloseTimeout": "10s",
        "attempt": 1
      }
    },
        {
      "eventId": "9",
      "eventTime": "2025-10-09T12:42:08.039527045Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_STARTED",
      "taskId": "2928672929",
      "workflowTaskStartedEventAttributes": {
        "scheduledEventId": "8",
        "identity": "12112@tmp-kt",
        "requestId": "6f02fe0a-cc3a-4208-bb93-0a11eb104e35",
        "historySizeBytes": "1196"
      }
    },
    {
      "eventId": "10",
      "eventTime": "2025-10-09T12:42:08.120574717Z",
      "eventType": "EVENT_TYPE_WORKFLOW_TASK_COMPLETED",
      "taskId": "2928672939",
      "workflowTaskCompletedEventAttributes": {
        "scheduledEventId": "8",
        "startedEventId": "9",
        "identity": "12112@tmp-kt",
        "workerVersion": {
          "buildId": "14de874560c3a2202fb30bb2b7483138"
        },
        "sdkMetadata": {},
        "meteringMetadata": {}
      }
    },
    {
      "eventId": "11",
      "eventTime": "2025-10-09T12:42:08.120635997Z",
      "eventType": "EVENT_TYPE_WORKFLOW_EXECUTION_COMPLETED",
      "taskId": "2928672940",
      "workflowExecutionCompletedEventAttributes": {
        "result": {
          "payloads": [
            {
              "metadata": {
                "encoding": "YmluYXJ5L251bGw="
              },
              "data": null
            }
          ]
        },
        "workflowTaskCompletedEventId": "10"
      }
    }
  ]
}