Single workflow schedule-to-start latency is within low hundreds of milliseconds

Community Support
, ,
1

Hi Temporal experts,

I’m seeking some guidance on “schedule-to-start” latency in my Temporal setup. I’m running experiments with workflows that only involve local activities (similar to the Eager Workflows use case). However, even with just a single workflow at a time, I’m observing a schedule-to-start latency around 200ms, and sync-match-latency around 100ms.

Here’s some additional context:

  • Temporal Version: 1.6.3 (managed, on-premises)
  • Deployment: Local Linux machines (client code and server on same machines)
  • Persistence: PostgreSQL
  • SDK Metrics: Persistence layer average latency ~20ms
  • PostgreSQL Query Analytics: Individual insert operations take ~3ms

My questions are:

  1. Why might the schedule-to-start latency be so high for a single workflow with local activities (considering the low persistence layer latency)?
  2. Could you explain how the SDK persistence layer latency metrics are calculated?

Any insights or suggestions to improve the performance would be greatly appreciated.

Thanks!

2

Hi @maxim / @tihomir sorry for directly tagging you guys, but this question is kind of time sensitive for me, so wanted to hear some of your thoughts.

Is there a way to achieve tens of milliseconds latency for schedule-to-start with an older temporal version (1.6.3)? The experiment I’m running is to start a short-living workflow every once in a while, so I’m wondering is there a reason why the matching step takes 200ms, assuming almost all my workers are free at the time when workflow task is pushed to Workflow task queue.