# Workflow task timed out on GKE **URL:** \ **Category:** Community Support\ **Tags:** java-sdk, cassandra, metrics\ **Created:** [June 7, 2022, 3:14pm UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909 "2022-06-07T15:14:47Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ruchir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ruchir/32/6141_2.png) [@Ruchir](https://community.temporal.io/u/Ruchir)\ **Post date:** [June 7, 2022, 3:14pm UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909/1 "2022-06-07T15:14:48Z")
Hi, Following is my temporal setup on GKE single node cluster. 1. Temporal cluster pointing to REMOTE Cassandra DB(TLS) 2. NoHistoryShards changed to 32 3. Match read/write partitions - 8 4. maxConns(Temporal): 200 5. Activity pollers: 50 6. Workflow pollers: 50 7. Frontend.NamespaceCount: 3000 8. MAX\_CONC\_WRK\_TASK\_EXEC\_SIZE: 200 9. MAX\_CONC\_ACT\_EXEC\_SIZE: 200 My workflow consist of 3 activities which makes http calls to microservices present in same cluster(assume latency of 40ms) Following were my readings for while running a load of 50 TPS for 1 min: avg=235.59ms min=215.6ms med=218.31ms max=520.25ms p(90)=317.25ms p(95)=319.59ms Following were my readings when i increase the TPS to 60: avg=3.09s min=215.34ms med=518.56ms max=11.06s p(90)=10.23s p(95)=10.25s Following are the metric graphs captured for 60 TPS: ![Screenshot 2022-06-07 at 8.38.59 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/a/a1e72c92679afde64d0bd12f814fdfc64b333c54.png) ![Screenshot 2022-06-07 at 8.39.34 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/2/257e014a0a022038778b1ee587d89858b6f1c9bd.png) ![Screenshot 2022-06-07 at 8.40.55 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/7/704a767059b4910c904f7704a287bdd75836882c.png) ![Screenshot 2022-06-07 at 8.41.31 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/8/8e396c8a9695ec133c598dd8e62467027a68ac82.png) ![Screenshot 2022-06-07 at 8.42.16 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/0/05cec793a3616b47d611cd6649fc68158c36de12.png) ![Screenshot 2022-06-07 at 8.32.40 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/0/05f092df53435811c008bf7b840a3bdd32e02472.png) As you can see temporal\_workflow\_task\_schedule\_to\_start\_latency\_seconds\_bucket, temporal\_workflow\_task\_execution\_latency\_seconds\_sum, temporal\_workflow\_endtoend\_latency\_seconds\_bucket, temporal\_activity\_schedule\_to\_start\_latency\_seconds\_sum, temporal\_activity\_execution\_latency\_seconds\_sum are very low. But still as you see in web ui, all the workflow tasks are getting timed out. Can you please help? ---
**Author:** ![tihomir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/tihomir/32/6580_2.png) [@tihomir](https://community.temporal.io/u/tihomir)\ **Post date:** [June 8, 2022, 4:18am UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909/3 "2022-06-08T04:18:38Z")
Could you provide full history for your execution: `tctl wf show -w -r --output_filename myhistory.json` Did you have the chance look through the [worker tuning guide in docs](https://docs.temporal.io/operation/how-to-tune-workers/)? Could you provide info on your sync match rate: ```auto sum(rate(poll_success_sync{}[1m])) / sum(rate(poll_success{}[1m])) ``` ideally it should be above 99%. if sync match rate is low it would mean your workers are unable to keep up (need to increase worker capacity) another thing to look at is sdk `task_schedule_to_start_latency` metric, can you measure this latency as well? a high latency would indicate to add more workers. > 1. NoHistoryShards changed to 32 i think this is too low, typically you would go with 512 for a small scale setup. for prod setup would start with 4K. Another thing to look at are persistence latencies: `histogram_quantile(0.95, sum(rate(persistence_latency_bucket{}[1m])) by (operation, le))` for operations: `CreateWorkflowExecution`, `UpdateWorkflowExecution`, `UpdateShard` > Following is my temporal setup on GKE single node cluster. How man instances of temporal services are you running on your test env? See [here](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/3) for recommendations for a prod setup ---
**Author:** ![Ruchir](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/ruchir/32/6141_2.png) [@Ruchir](https://community.temporal.io/u/Ruchir)\ **Post date:** [June 8, 2022, 7:33am UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909/4 "2022-06-08T07:33:53Z")
> [@tihomir](#): > > `tctl wf show -w -r --output_filename myhistory.json` Following is the required json output. Note that I have removed certain fields like input and results which has sensitive data: ```auto 1 WorkflowExecutionStarted {WorkflowType:{Name:IRequestOtpWorkflow}, ParentInitiatedEventId:0, TaskQueue:{Name:RequestOtpWorkflowTaskQueue, Kind:Normal}, Input:[], WorkflowExecutionTimeout:0s, WorkflowRunTimeout:0s, WorkflowTaskTimeout:10s, Initiator:Unspecified, OriginalExecutionRunId:66fa450b-ced0-4475-bc36-78728d7d8fb1, FirstExecutionRunId:66fa450b-ced0-4475-bc36-78728d7d8fb1, Attempt:1, FirstWorkflowTaskBackoff:0s, } 2 WorkflowTaskScheduled {TaskQueue:{Name:RequestOtpWorkflowTaskQueue, Kind:Normal}, StartToCloseTimeout:10s, Attempt:1} 3 WorkflowTaskStarted {ScheduledEventId:2, RequestId:fed0d40d-19d2-43e8-8d1c-33e58b82c568} 4 WorkflowTaskCompleted {ScheduledEventId:2, StartedEventId:3, 5 MarkerRecorded {MarkerName:LocalActivity, Details:map{activityId:["08d4dbf4-366f-32e3-a5b6-80911d97ff20"], input:[{}], time:[1654609796207], type:["Validate"]}, WorkflowTaskCompletedEventId:4} 6 ActivityTaskScheduled {ActivityId:8a240c73-55c2-30ba-b0fe-22ee16c5090c, ActivityType:{Name:GenerateOTP}, TaskQueue:{Name:OtpProcessorActivityTaskQueue, Kind:Normal}, ScheduleToCloseTimeout:0s, ScheduleToStartTimeout:0s, StartToCloseTimeout:5s, HeartbeatTimeout:0s, WorkflowTaskCompletedEventId:4, RetryPolicy:{InitialInterval:1s, BackoffCoefficient:2, MaximumInterval:1m40s, MaximumAttempts:1, NonRetryableErrorTypes:[com.fyndna.common.type.CustomCheckedException]}} 7 ActivityTaskStarted {ScheduledEventId:6, RequestId:8d92c17a-46cf-4d15-91be-d84e9c679b9e, Attempt:1} 8 ActivityTaskCompleted {Result:[{"errorDetail":"Mock Adapter","refNo":"02032016344072","statusCode":"00"}], ScheduledEventId:6, StartedEventId:7, 9 WorkflowTaskScheduled {TaskQueue:{Name:1@issuer-switch-upi-registration-switch-66db64df-jw5c6:28f2f188-f52f-4129-b21b-06b3d2076318, Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1} 10 WorkflowTaskTimedOut {ScheduledEventId:9, StartedEventId:0, TimeoutType:ScheduleToStart} 11 WorkflowTaskScheduled {TaskQueue:{Name:RequestOtpWorkflowTaskQueue, Kind:Normal}, StartToCloseTimeout:10s, Attempt:1} 12 WorkflowTaskStarted {ScheduledEventId:11, RequestId:2fd8d1cb-ab48-4a7b-a38b-e748b2e544b2} 13 WorkflowTaskCompleted {ScheduledEventId:11, StartedEventId:12, 14 ActivityTaskScheduled {ActivityId:a6cf7ea8-9f4a-3430-bed3-ac612ba5a95b, ActivityType:{Name:DeliverOtp}, TaskQueue:{Name:OtpCommunicationActivityTaskQueue, Kind:Normal}, ScheduleToCloseTimeout:0s, ScheduleToStartTimeout:0s, StartToCloseTimeout:5s, HeartbeatTimeout:0s, WorkflowTaskCompletedEventId:13, RetryPolicy:{InitialInterval:1s, BackoffCoefficient:2, MaximumInterval:1m40s, MaximumAttempts:1, NonRetryableErrorTypes:[com.fyndna.common.type.CustomCheckedException]}} 15 ActivityTaskStarted {ScheduledEventId:14, RequestId:b0b92566-d116-429f-ba5f-792df6251738, Attempt:1} 16 ActivityTaskCompleted {Result:[{"responseCode":"00","message":"Success"}], ScheduledEventId:14, StartedEventId:15, 17 WorkflowTaskScheduled {TaskQueue:{Name:1@issuer-switch-upi-registration-switch-66db64df-jw5c6:28f2f188-f52f-4129-b21b-06b3d2076318, Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1} 18 WorkflowTaskTimedOut {ScheduledEventId:17, StartedEventId:0, TimeoutType:ScheduleToStart} 19 WorkflowTaskScheduled {TaskQueue:{Name:RequestOtpWorkflowTaskQueue, Kind:Normal}, StartToCloseTimeout:10s, Attempt:1} 20 WorkflowTaskStarted {ScheduledEventId:19, RequestId:47cd83bf-270e-4f2f-8e03-f78ebc7cc6af} 21 WorkflowTaskCompleted {ScheduledEventId:19, StartedEventId:20, 22 ActivityTaskScheduled {ActivityId:fc7968a0-1fd7-3271-ad20-b5f9cadf8eb5, ActivityType:{Name:SendOtpResponse}, TaskQueue:{Name:RequestOtpActivityTaskQueue, Kind:Normal},, Input:[], ScheduleToCloseTimeout:0s, ScheduleToStartTimeout:0s, StartToCloseTimeout:5s, HeartbeatTimeout:0s, WorkflowTaskCompletedEventId:21, RetryPolicy:{InitialInterval:1s, BackoffCoefficient:2, MaximumInterval:1m40s, MaximumAttempts:1, NonRetryableErrorTypes:[]}} 23 ActivityTaskStarted {ScheduledEventId:22, RequestId:d0e106e2-0b3e-4d8f-bd74-11f57af91d70, Attempt:1} 24 ActivityTaskCompleted {Result:[], ScheduledEventId:22, StartedEventId:23, 25 WorkflowTaskScheduled {TaskQueue:{Name:1@issuer-switch-upi-registration-switch-66db64df-jw5c6:28f2f188-f52f-4129-b21b-06b3d2076318, Kind:Sticky}, StartToCloseTimeout:10s, Attempt:1} 26 WorkflowTaskTimedOut {ScheduledEventId:25, StartedEventId:0, TimeoutType:ScheduleToStart} 27 WorkflowTaskScheduled {TaskQueue:{Name:RequestOtpWorkflowTaskQueue, Kind:Normal}, StartToCloseTimeout:10s, Attempt:1} 28 WorkflowTaskStarted {ScheduledEventId:27, RequestId:1a2fdd82-1012-4edd-916a-50a82fa54e6e} 29 WorkflowTaskCompleted {ScheduledEventId:27, StartedEventId:28, 30 WorkflowExecutionCompleted {Result:[{}], WorkflowTaskCompletedEventId:29} ``` > [@tihomir](#): > > Did you have the chance look through the [worker tuning guide in docs](https://docs.temporal.io/operation/how-to-tune-workers/?_ga=2.185391585.99919015.1654485373-1781121356.1643869734)? Yes Tihomir, we have gone through this link and we had tried for different configs, but whenever we go above 50 tps, we start to see very high latency. > [@tihomir](#): > > `sum(rate(poll_success_sync{}[1m])) / sum(rate(poll_success{}[1m]))` ![Screenshot 2022-06-08 at 12.31.52 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/5/50781e86fb5e3ed5ce05d4dfaa3fb970860a3828.png) > [@tihomir](#): > > another thing to look at is sdk `task_schedule_to_start_latency` metric, can you measure this latency as well? a high latency would indicate to add more workers. Uploading: Screenshot 2022-06-08 at 12.33.55 PM.png… When you say “add more workers/ increase worker capacity”, does that mean increasing concurrent pollers/ increasing execution size/ increasing pods of application where we create workers or something else? > [@tihomir](#): > > i think this is too low, typically you would go with 512 for a small scale setup. for prod setup would start with 4K. We had tried this with 512 shards, and had observed low latency with default matching partitions, but as we try the same run with 8 matching service partitions, the latency again increases. We want to understand why this is happening. > [@tihomir](#): > > `histogram_quantile(0.95, sum(rate(persistence_latency_bucket{}[1m])) by (operation, le))` > for operations: `CreateWorkflowExecution`, `UpdateWorkflowExecution`, `UpdateShard` ![Screenshot 2022-06-08 at 12.38.35 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/9/984aceffe7ed93ee765e481cc0835e458ec666c1.png) ![Screenshot 2022-06-08 at 12.38.56 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/a/a4b86bcdd1556d3c6c6cde500a09db5189cd3af0.png) ![Screenshot 2022-06-08 at 12.39.59 PM](https://us1.discourse-cdn.com/flex016/uploads/temporal/original/2X/7/72adde51a44f121f6301947df2e22999c20c55a8.png) > [@tihomir](#): > > How man instances of temporal services are you running on your test env? See [here](https://community.temporal.io/t/what-is-the-recommended-setup-for-running-cadence-temporal-with-cassandra-on-production/556/3) for recommendations for a prod setup We have gone through this link as well. Currently there are only 1 pod per temporal service. We wanted to see with this deployment how much TPS can be achieved. ---
**Author:** ![poojabhutada](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/poojabhutada/32/3372_2.png) [@poojabhutada](https://community.temporal.io/u/poojabhutada)\ **Post date:** [June 8, 2022, 4:35pm UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909/5 "2022-06-08T16:35:19Z")
@tihomir Also, some followup questions regarding above setup. 1. We were able to achieve decent response time of workflow execution till 60 tps…but it goes very worse above that despite of changing no of shards, increasing temporal services pods, increasing no. of partitions. No luck yet. 2. Another thing, looks like when it goes above 60 tps…workflow active thread count drastically increases, sync match rate drops and the service latency that invokes async workflow execution increases. Is there any parameter to check the latency of a workflow invocation? Please do help and guide us. ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [June 8, 2022, 5:20pm UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909/6 "2022-06-08T17:20:34Z")
> 1. NoHistoryShards changed to 32 > 2. Match read/write partitions - 8 > 3. maxConns(Temporal): 200 try not to use single k8s pod, but one pod for frontend / matching / history try 256 as history shards, 1 read / write partition (matching) what is `maxConns`? * * * how many workflows are running concurrently? ---
**Author:** ![poojabhutada](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/poojabhutada/32/3372_2.png) [@poojabhutada](https://community.temporal.io/u/poojabhutada)\ **Post date:** [June 8, 2022, 5:44pm UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909/7 "2022-06-08T17:44:45Z")
@Wenquan_Xing We do have one pod each per temporal service i.e frontend, history, matching etc MaxConns is defined as 200…it is the value provided as part of yaml file for default and visibility databases…any recommendation you have for this value? We are trying to hit workflows around 70 TPS…our target is to achieve more than 3000 TPS. ---
**Author:** ![Wenquan\_Xing](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/wenquan_xing/32/362_2.png) [@Wenquan\_Xing](https://community.temporal.io/u/Wenquan_Xing)\ **Post date:** [June 8, 2022, 5:50pm UTC](https://community.temporal.io/t/workflow-task-timed-out-on-gke/4909/8 "2022-06-08T17:50:38Z")
> [@poojabhutada](#): > > MaxConns is defined as 200 1. are you using advance visibility? 2. `MaxConns is defined as 200`, try 8? > We are trying to hit workflows around 70 TPS…our target is to achieve more than 3000 TPS. TPS: what is your definition? 70 workflow completion per second? a workflow with 3 sequential activities roughly translate to 15 DB transactions you may want to check if your DB is overloaded already