Workflow Timeouts with Activities Remaining in Scheduled State Under High Load

[Abhinav_Verma]

Hi everyone,

I’m investigating a large number of workflow timeouts during a load test and would appreciate some guidance.

Setup

Worker configuration:

{
  "maxConcurrentWorkflowTaskPollers": 1,
  "maxConcurrentActivityTaskPollers": 1,
  "maxConcurrentWorkflowTaskExecutionSize": 300,
  "maxConcurrentActivityExecutionSize": 300
}

Traffic

During the test:

• ~360,000 requests over 5 minutes in 17 batches
• ~303,000 workflows timed out

What I observe

In Temporal Web:

• The workflow ends with WorkflowExecutionTimedOut
• Several activities remain in PENDING_ACTIVITY_STATE_SCHEDULED
• These activities have:
  • Attempt = 1
  • No ActivityTaskStarted event
  • No ActivityTaskTimedOut event

Example timeline:

ActivityTaskScheduled
ActivityTaskScheduled
ActivityTaskScheduled
ActivityTaskScheduled

WorkflowExecutionTimedOut

The activities never seem to reach the Started state before the workflow times out.

My Questions

1. Does this indicate worker saturation / insufficient activity consumption capacity?
2. Could maxConcurrentActivityTaskPollers=1 and maxConcurrentWorkflowTaskPollers=1 become a bottleneck at this scale?
3. Are there any Temporal server-side metrics that would clearly show whether tasks were sitting in the queue waiting for workers?

My current hypothesis is that activities were being scheduled faster than workers could pick them up, causing workflows to hit their execution timeout while activities were still waiting in the task queue, but I’d like to validate whether that’s a reasonable conclusion.

Thanks!

[Abhinav_Verma]

@maxim @tihomir Can you guys help ??