# High latency on Starting Workflow/Activity execution **URL:** \ **Category:** Community Support\ **Tags:** java-sdk, metrics, latency\ **Created:** [March 1, 2023, 4:11pm UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439 "2023-03-01T16:11:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Dhanraj](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/dhanraj/32/1790_2.png) [@Dhanraj](https://community.temporal.io/u/Dhanraj)\ **Post date:** [March 1, 2023, 4:11pm UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439/1 "2023-03-01T16:11:17Z")
Hi Team, We have developed basic POC app that orchestrates between activities(11 in total). These activities are one liners and only logs a simple message to console. We deployed this app to azure-westeurope region in k8s cluster close to our self hosted temporal cluster(westeurope). Upon executing the requests, we could see the entire **workflow taking close to 3sec** to complete execution. This is very high latency considering simple use case as this. The same application in non-temporal env takes less than 100ms to execute. We tried Local Activity and workflow could complete execution in 400ms. This will be our last resort as we plan to have dedicated workers for activities & leverage rate-limiting. We enabled sdk tracing and this is how it looks: ![Screenshot 2023-03-01 at 8.57.15 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/3/371eb9d53d155dd4d6503d7271cb87145f2143f1.jpeg) As we can see from traces, most of the **time is spent on Starting workflow & activity execution**. **Our CPU & Mem where all normal, that rules out worker is overutilized** ![Screenshot 2023-03-01 at 9.18.11 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/9/9f6251311a73cd5ce5636ccc6057f5dbbb647e4d.png) ![Screenshot 2023-03-01 at 9.18.18 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/3/38ab5fc661160e2f11df266b76450787ee4b9324.png) Below are workflow/activity schedule\_to\_start latencies & grpc request\_latency ![Screenshot 2023-03-01 at 8.55.42 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/4/4a73093bd8a1a6bf41904398025f4e99e34a9d5e.png) ![Screenshot 2023-03-01 at 8.56.51 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/c/c1d2532d90f2ab51bdb2bf2fc61c399aec9d9911.png) We could see schedule\_to\_start is comparatively low than request\_latency. The request\_latency is on par with overall latencies that we observed on traces. **How schedule\_to\_start latency is measured? Does request\_latency also included schedule\_to\_start interval?** We also tried to increase **setMaxConcurrentActivityTaskPollers** from default 5, but that didn’t yield any difference in response latencies. **Temporal server version - 1.18.4** **Java SDK version - 1.18** Could you please help troubleshoot this issue? Thanks, Dhanraj ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [March 1, 2023, 7:42pm UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439/3 "2023-03-01T19:42:56Z")
> How schedule\_to\_start latency is measured? This is measured from the time a workflow/activity task is placed on a matching host task queue partition until your worker picks it up for processing. > Does request\_latency also included schedule\_to\_start interval? request\_latency long\_request\_latency measure only the gRPC level latencies so how long client/worker requests themselves take. Schedule to start latencies deal with how long it takes for tasks to be dispatched to your workers. ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [March 1, 2023, 7:48pm UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439/4 "2023-03-01T19:48:28Z")
Regarding your rpc latencies for StartWorkflowExecution, can you share your persistence latencies (Server metrics): `histogram_quantile(0.95, sum(rate(persistence_latency_bucket{}[1m])) by (operation, le))` And frontend service latencies (server metrics): `histogram_quantile(0.95, sum(rate(service_latency_bucket{service_type="frontend"}[5m])) by (operation, le))` > We also tried to increase **setMaxConcurrentActivityTaskPollers** from default 5, but that didn’t yield any difference in response latencies. Share sync match rate please: `sum(rate(poll_success_sync{}[1m])) / sum(rate(poll_success{}[1m]))` ---
**Author:** ![Dhanraj](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/dhanraj/32/1790_2.png) [@Dhanraj](https://community.temporal.io/u/Dhanraj)\ **Post date:** [March 2, 2023, 12:46pm UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439/5 "2023-03-02T12:46:29Z")
Hi @tihomir , Please find query results for the metrics that you asked **Persistence latencies** ![Screenshot 2023-03-02 at 6.00.50 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/a/acde2bb82423fb35e496ad000ac0e7d23ff1c839.jpeg) ![Screenshot 2023-03-02 at 6.11.57 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/e/e51d03c6882d7f4efb4797f118f658c74e8fe763.png) **Frontend service latencies** ![Screenshot 2023-03-02 at 6.00.01 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/1/11ad9d35ae26748f85a3390f77b9cf09bf3482c0.png) ![Screenshot 2023-03-02 at 5.59.43 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/a/a5fdde3d7d2d691c68182b3ed47f8a56be3efbd5.png) **Sync match rate** ![Screenshot 2023-03-02 at 6.01.26 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/f/fcf54692650077e1841a26b809acbd097cb9b80e.png) Please let me know if you need more info. @Andrey_Dubnik - fyi Thanks, Dhanraj ---
**Author:** ![Andrey\_Dubnik](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/andrey_dubnik/32/1498_2.png) [@Andrey\_Dubnik](https://community.temporal.io/u/Andrey_Dubnik)\ **Post date:** [March 4, 2023, 9:29pm UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439/6 "2023-03-04T21:29:00Z")
We would be really grateful if someone can help us to understand what does StartActivity and RunActivity in the distributed trace corresponds to. If StartActivity is the schedule to start and Run Activity is the execution duration then the there is an unexplained gap between the end of the StartActivity and RunActivity measurements. ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [March 5, 2023, 5:50am UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439/7 "2023-03-05T05:50:14Z")
@Dhanraj Poll latency being high is something expected, if its low it means your workers rarely need to wait for tasks (meaning you probably don’t have enough workers) PollWorkflowExecutionHistory is a long poll request too so it being high is imo ok too. Regarding high latencies for StartWorkflowExecution , can you check if your frontend and/or history hosts have high cpu use? ---
**Author:** ![Andrey\_Dubnik](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/andrey_dubnik/32/1498_2.png) [@Andrey\_Dubnik](https://community.temporal.io/u/Andrey_Dubnik)\ **Post date:** [March 6, 2023, 5:19pm UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439/8 "2023-03-06T17:19:57Z")
There is no load on frontend or history. ---
**Author:** ![Dhanraj](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/dhanraj/32/1790_2.png) [@Dhanraj](https://community.temporal.io/u/Dhanraj)\ **Post date:** [March 15, 2023, 1:17pm UTC](https://community.temporal.io/t/high-latency-on-starting-workflow-activity-execution/7439/9 "2023-03-15T13:17:13Z")
@tihomir - As Andrey mentioned, there was no load on our Frontend & History hosts. Could you please help us troubleshoot this issue? Please let me know if you need any details/metrics.