Temporal workflow has blank delays/gaps between activities in local/cloud setup. Is this expected?

Community Support
1

Expected Behavior

After an activity completes, the next activity in the workflow sequence should be scheduled promptly — with only minimal overhead (milliseconds) between ActivityTaskCompleted and the next ActivityTaskScheduled event in the workflow history. In a healthy Temporal setup, the Workflow Task that drives the next execute_activity call should be picked up and executed quickly, even under moderate concurrency with child workflows.

Actual Behavior

Image

A blank/invisible gap of 10–20 seconds appears between consecutive activities in the same ChildWorkflow. Specifically:

  • An activity (e.g. predict_feedback_reference) completes successfully

  • No next activity is scheduled for 10–20 seconds

  • Only after that gap does the next activity (e.g. predict_transaction_merchant) appear as ActivityTaskScheduled

This gap occurs before the next activity is scheduled — meaning the delay is in Workflow Task pickup/execution, not in the activity queue or the activity execution itself. This is observed in both local Docker and Temporal Cloud environments, across a workload of ~50–150 concurrent child workflow executions.

Steps to Reproduce the Problem

  1. Run a Temporal server locally via Docker with constrained resources (cpus: 0.5, mem_limit: 1024m)

  2. Start a single worker process with MAX_CONCURRENT_WORKFLOW_TASKS=100 and MAX_CONCURRENT_ACTIVITIES=100 on the same machine

  3. Trigger a parent workflow that fans out to 50–150 child workflows, each executing 4 sequential activities via await workflow.execute_activity(...)

  4. Open the Temporal UI and inspect the event history of any child workflow

  5. Observe the timestamp gap between ActivityTaskCompleted and the subsequent ActivityTaskScheduled — expect ~10–20 second blank gaps

Specifications

Field Value
Temporal Python SDK Version 1.x (latest)
Temporal Server temporalio/auto-setup:latest (Docker)
Temporal Server Resources 0.5 CPUs, 1024 MB RAM, 512 PIDs
Platform / OS macOS (worker + Docker host)
Observed On Local Docker + Temporal Cloud
Worker Concurrency MAX_CONCURRENT_WORKFLOW_TASKS=100, MAX_CONCURRENT_ACTIVITIES=100
Worker Processes 1
Child Workflow Count ~50–150 concurrent
Database Backend PostgreSQL 12
2

Troubleshooting and resolving bottlenecks on both server and sdk worker side is pretty useful and common practice.
Often times these types of schedule-to-start latencies could have different and multiple underlying causes, things like

  • small / slow / overloaded db
  • small / unprovisioned cluster (for example very small default number of shards)
  • resource starvation on cluster side
  • unprovisioned / resource starved workers
  • others

from your description, one thing that is fairly obvious is the cluster side, you run auto-setup image which runs all 4 temporal services in single process on pretty low compute, and locally on your laptop it seems. we dont really know about db resources given as assume it runs locally as well

this in itself is probably a possible bottleneck where some latencies are expected when running even smaller size fan-out use cases

one thing that could help to troubleshoot is server and your worker metrics
you have out of box grafana dashboards for both here, and here, if helps

if your trying to run and test locally on docker desktop that seems to be case, maybe look at this repo as well, as it would allow you to scale out better than using single process on auto-setup image
as well as it has built in observability (prometheus and grafana) so you dont have to set that up yourself. its till not meant for prod testing, but maybe would give you env where you can get some base lines better

if you rather run your cluster locally on docker desktop and kubernetes, check out this repo if helps