Huge latency overhead on very simple setup

Community Support
,
1

We are testing our “real-world” case for workflow (see screenshot).

wf_chart
wf_chart2682×748 151 KB

The setup is:

  1. Python SDK
  2. One workflow, one task queue
  3. Single worker pod with all the default values
  4. Absolutely no load (just single runs)

We run it multiple times and always get pretty much the same result - huge latency overhead on the temporal server side - big gap on the first activity run, and also a noticable gaps between activities. Also, some activities have gap between scheduling and start.

Doesnt look like a normal behavior.

Can anyone suggest on how to debug/tune this setup to reduce those gaps?

Thank you.

2

Can you provide a small, standalone replication? Maybe adjust one of the Python samples to show what you are seeing? Are you perhaps doing something to affect the Python asyncio event loop like doing thread-blocking calls in an async function? Also note the axis on the graph there, the latencies are in milliseconds not seconds, but still probably too high, just make sure that nothing else on the system is interfering and there are no network issues.