Temporal workflow has blank delays/gaps between activities in local/cloud setup. Is this expected?

Community Support
1

Expected Behavior

After an activity completes, the next activity in the workflow sequence should be scheduled promptly — with only minimal overhead (milliseconds) between ActivityTaskCompleted and the next ActivityTaskScheduled event in the workflow history. In a healthy Temporal setup, the Workflow Task that drives the next execute_activity call should be picked up and executed quickly, even under moderate concurrency with child workflows.

Actual Behavior

Image

A blank/invisible gap of 10–20 seconds appears between consecutive activities in the same ChildWorkflow. Specifically:

  • An activity (e.g. predict_feedback_reference) completes successfully

  • No next activity is scheduled for 10–20 seconds

  • Only after that gap does the next activity (e.g. predict_transaction_merchant) appear as ActivityTaskScheduled

This gap occurs before the next activity is scheduled — meaning the delay is in Workflow Task pickup/execution, not in the activity queue or the activity execution itself. This is observed in both local Docker and Temporal Cloud environments, across a workload of ~50–150 concurrent child workflow executions.

Steps to Reproduce the Problem

  1. Run a Temporal server locally via Docker with constrained resources (cpus: 0.5, mem_limit: 1024m)

  2. Start a single worker process with MAX_CONCURRENT_WORKFLOW_TASKS=100 and MAX_CONCURRENT_ACTIVITIES=100 on the same machine

  3. Trigger a parent workflow that fans out to 50–150 child workflows, each executing 4 sequential activities via await workflow.execute_activity(...)

  4. Open the Temporal UI and inspect the event history of any child workflow

  5. Observe the timestamp gap between ActivityTaskCompleted and the subsequent ActivityTaskScheduled — expect ~10–20 second blank gaps

Specifications

Field Value
Temporal Python SDK Version 1.x (latest)
Temporal Server temporalio/auto-setup:latest (Docker)
Temporal Server Resources 0.5 CPUs, 1024 MB RAM, 512 PIDs
Platform / OS macOS (worker + Docker host)
Observed On Local Docker + Temporal Cloud
Worker Concurrency MAX_CONCURRENT_WORKFLOW_TASKS=100, MAX_CONCURRENT_ACTIVITIES=100
Worker Processes 1
Child Workflow Count ~50–150 concurrent
Database Backend PostgreSQL 12