Workflow execution taking ~2 minutes even though activities are fast

Community Support
1

Problem summary

  • All activities complete in milliseconds

  • No retries, no Workflow.sleep, no awaits/signals

  • But the overall workflow execution took ~2 minutes

  • The delay happens before the first workflow task starts

Evidence from Event History

From the workflow event history:

WORKFLOW_EXECUTION_STARTED   @ 23:22:13
WORKFLOW_TASK_SCHEDULED      @ 23:22:13
WORKFLOW_TASK_STARTED        @ 23:24:31

Schedule-to-start latency ~138 seconds for the FIRST workflow task

After this point:

  • Workflow tasks are picked up immediately

  • Activities execute in milliseconds

  • No further gaps or delays

This suggests the time was spent waiting for a workflow task to be picked up, not executing workflow or activity code.

Worker configuration (IssueBundle workflow)

{
  "workerName": "IB1",
  "taskQueue": "IBTasksQueue",
  "maxConcurrentWorkflowTaskPollers": 50,
  "maxConcurrentActivityTaskPollers": 50,
  "maxConcurrentWorkflowTaskExecutionSize": 300,
  "maxConcurrentActivityExecutionSize": 300
}

What are the common causes of large workflow task schedule-to-start latency like this?

What is the best way to debug this further?

  • Which Temporal metrics should I rely on?

  • What logs or events are most useful?

  • Any specific worker-side instrumentation recommended?

2

image
image1840×1250 179 KB

image
image1900×1040 113 KB

image
image1832×1104 142 KB

Putting up some grafana metrics for better understanding.