# What factors count for Workflow Schedule to Start Latency?

**URL:** <https://community.temporal.io/t/what-factors-count-for-workflow-schedule-to-start-latency/5829>\
**Category:** Community Support\
**Tags:** performance\
**Created:** [August 29, 2022, 1:15am UTC](https://community.temporal.io/t/what-factors-count-for-workflow-schedule-to-start-latency/5829 "2022-08-29T01:15:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![whitecrow](https://avatars.discourse-cdn.com/v4/letter/w/50afbb/32.png) [@whitecrow](https://community.temporal.io/u/whitecrow)\
**Post date:** [August 29, 2022, 1:15am UTC](https://community.temporal.io/t/what-factors-count-for-workflow-schedule-to-start-latency/5829/1 "2022-08-29T01:15:58Z")

</div>

I did a performance test on a CentOS system with 8 processors and 64 GB memory.  
My test case is a simple workflow which contains a single trivial activity.

I found with the concurrency raised up, the throughput hit a limit, around 100 executions/second. A large workflow schedule to start latency was observed, and it increased with the number of concurrent requests， as shown below:

 ![截屏2022-08-28 下午5.06.40](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/9/9a4b2e1812438285b5a26204f3e9c94182723756.png)  
On the other hand, the latency of activity looks acceptable.  
 ![截屏2022-08-29 上午9.04.31](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/2/20ccbb0f9f9305e92fc6236332adecf3df58536e.png)

I confirmed that all the machine resources did not hit their limits, including cpu/memory/diskIO. According to the CPU usages, it seems none of the temporal components overloaded.

 ![截屏2022-08-26 下午3.37.02](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/0/02e3b6a699776f32569e3ba65ca7f50a4b067478.png)

And I tried tuning the parameters mentioned [here](https://community.temporal.io/t/tuning-temporal-setup-for-better-performance/539/5), but none of them made much difference.

So I want to know what factors affect the Workflow Schedule to Start Latency. Any suggestions that may help to find the bottleneck in the test above would be appreciated.

---

<div class="post-metadata">

**Author:** ![spikhalskiy](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/spikhalskiy/32/1250_2.png) [@spikhalskiy](https://community.temporal.io/u/spikhalskiy)\
**Post date:** [August 29, 2022, 1:53am UTC](https://community.temporal.io/t/what-factors-count-for-workflow-schedule-to-start-latency/5829/2 "2022-08-29T01:53:12Z")

</div>

> And I tried tuning the parameters mentioned [here](https://community.temporal.io/t/tuning-temporal-setup-for-better-performance/539/5), but none of them made much difference.

You can follow a full [Workers tuning guide](https://docs.temporal.io/application-development/worker-performance/) for step by step walk through parameters and the corresponding metrics to check from the Worker side.

I would totally expect the default Worker config to start bump into the `maxConcurrentWorkflowTaskExecutionSize` value on 100+ workflows/second if you have one worker only. So I would point attention to this parameter and the corresponded `worker_task_slots_available` metric value.

From a server side somebody else will help better, but it’s a good idea to check db latency metrics.

---

<div class="post-metadata">

**Author:** ![whitecrow](https://avatars.discourse-cdn.com/v4/letter/w/50afbb/32.png) [@whitecrow](https://community.temporal.io/u/whitecrow)\
**Post date:** [August 29, 2022, 2:14am UTC](https://community.temporal.io/t/what-factors-count-for-workflow-schedule-to-start-latency/5829/3 "2022-08-29T02:14:30Z")

</div>

Sure, I will check the metrics you mentioned.  
More informations about my test:  
I used to upgraded my Cassandra DB from 3.11 to 4.0.6, and 2x throughput was obtained. But the behavior pattern does not change.  
I tried creating more duplicates of each temporal components, but did not make much improvements.